Development of the full stack web application “Commodoro”
Study case of the development of Commodoro the largest network of coffee farmers
The following blog will share the experience of the development of a full stack web application since the conception of the idea, the purpose of the web app, to the stack chosen, partial algorithm, process development and the next milestones to be achieved.
Also, this blog is for those who are interested in the development of technological sustainable products as well as to see from a larger perspective how an app is the sum of other non-technical factors such as the election and efficient team work, the design and development of an organized project, the leadership of each member of the team and other important agile skills in order the continuity and success of an app can be traceable and can be functional applied in the market.
The Conception
It is said that everything begins with an idea. In fact, in this case it was an idea shared not by a single individual but from three ambitious programmers that saw a way to help a larger community of people who sometimes is relegated by society and it is not rewarded properly for the huge contribution to society. This community are the farmers. They are in large part responsible for harvest the food we eat daily. We, Edward Ortiz, Carlos Molano and David Peralta, shared this concern and despite the three of us are full stack developers, we also saw in each other a different background in electrical engineering, project management, food management and skills that when merged into a single idea, the result was the development of a product thinking for this community and therefore something that will gain traction to farmers and boost the impact of agriculture with applied technology to support one of many today’s world challenge.
The Idea
As mentioned before, we were concerned about the hard work the farmers have today and mostly that they struggle to sell their product and make a sustainable profit. People hear often that production cost in agriculture exceed the revenue of the farmer thus causing a deficit; either for climate change or for a decreasing in the demand of that product, the current economic market doesn’t have enough benefits to cover cost overrun of the production of a harvest. This is a big problem and farmers due this conditions they have to sell their product either for private companies or government institutions so the price traded of the goods and based in a centralized market exchange.
the current economic market doesn’t have enough benefits to cover cost overrun of the production of a harvest.
However, when addressing this issue for the current agricultural system, we knew that in order to seek a solution we needed to close more the scope of the problem and take as a pivot, one agricultural product or as some people technically call these products (agricultural commodities), so we can develop a technological product that can be successful validated and replicated. We, knew that in the coffee market there has been a new market of trade in which private clients start a trade a process directly with the coffee farmer and trade the coffee for a fair price based in factors such as quality, organic harvesting, washing process, terrain specifications, social impact and more. Also by living in Colombia one of the largest producers of coffee in the world*, we took the time to understand the complete process for a farmer to sell their coffee either if is for national selling or exporting and to recognize that with over 560.000 coffee farms in Colombia, the country had a portion of USD$2.12B in the market over a total global of USD $34B annually in this industry. These facts among others, confirm us to create a solution starting with this commodity and to somehow create a practical solution not only for the 540.000 farmers in Colombia but for the other millions around the world.
Then, by pivoting the problem with solutions we agreed that a bottleneck for the farmers is that they always are in risk due the production cost vs the price they receive based on the market. Also, despite a new model of trade is increasing in Colombia (the one which farmer and seller is negotiating directly) not all the farmers are aware of that trade model, or cannot reach the seller for reasons such as their difficulty to leave the farm and attend international fairs, tradition in selling their product and the education given for marketing their coffee.
Therefore, we knew that by creating a product in which a farmer can be visible to clients without leaving the farm as well as to marketing their coffee, it will increase the opportunity to increase their sales or at least the presence of their farm in the market. Then, to achieve this we thought in the creation of a web application that is a professional community online for coffee farmers, where this community basically shares profiles of the farms with the characteristics of the product, history of the farm, and other relevant data that could engage the interest of a client to contact the farmer and to trade with fair price.
Finally, we conceived the product, so the next step was to name it and we thought about Commodoro “Commodore in English”, because this word had in the past a strong relationship with merchant ships and trading of several goods. We believed that by creating a solid community for farmers in general to be engaged with private clients they will gain traction to their products and transform their farms into more productive ones with a fair trade based in decentralized market model.
we knew that by creating a product in which a farmer can be visible to clients without leaving the farm as well as to marketing their coffee, it will increase the opportunity to increase their sales or at least the presence of their farm in the market.
The Design
The idea was defined, now the team had to assemble roles and develop our product as a technological sustainable solution. Initially, the team decided to separate each role based on the large scale of production that the application will have. This means that Commodoro will have a backend, frontend and infrastructure activity.
- Frontend: This activity is focused to the user experience and interface that Commodoro will implement. For this stage, the goal was to create a friendly user experience due most of the users will be farmers, then to be able to engage the farmers with an excellent usability as well as the supplier with the design of the site, was an objective to achieve in this area. David Peralta in this case assumed the role of solve this challenges and work as a leader in this stage.
- Backend: The scope of Commodoro is to evolve continuously as the users are interacting with the web app. Therefore, the design and programming of a robust backend system that will support and handle X times concurrent users and securing their data was the main goal for the development of this platform. Also, due the first deliverable is to create a custom profile page of the farms the team wanted to craft a method to handle automatic updates of the information provided by the user. Carlos Molano took the leap and assumed the role of backend leader.
- Infrastructure: Tied with the backend and frontend activities, the infrastructure is a key aspect for the platform. Many ideas came into mind by the time to deploy Commodoro as a functional product. LAMP stack is the most used and most reliable infrastructure for deploy an initial product, but considering the skills and experience of the team, we thought to implement another infrastructure based in the factor of concurrency of users and usage of each feature of commodore in a microservice infrastructure rather than a monolithic architecture.
By that moment we knew that there were many activities to do at the same time for develop the web application used trello, so we decided to use trello and fragment the activities in phases. The good thing about the tool trello is that a project allows to design practically the phases, roles and tasks of a project. Then the following image is a sample on how we gathered the tasks in the backend, frontend and infrastructure to have a better grasp and monitoring of the project development.
The Timeline
Commodoro team had a deadline of five weeks to deliver a MVP. However, we agreed to work over a six-week timeline in which the week 5 will be the presentation of the MVP and the week 6 the continuous improvement and discussion about the mutation of the project for another stage.
The following image present Commodoro timeline in this first stage:
The Stack
The stack chosen for Commodoro was an interesting discussion mostly because we as team wanted to challenge ourselves, and based in our skills and goals defined for Commodoro, the technological stack should be as reliable and secure as possible. For outline the stack we thought of two main actions that Commodoro will have:
- Concurrency in users and actions performed
- Microservice architecture
Based on these two factors, the implementation of Commodoro in a monolithic stack for example NodeJS in the backend and ReactJS in the fronted was out of discussion. So we knew that Python is a high-level, general-purpose programming language for develop the backend in order to ensure models management and future decentralized transactions in real-time.
To support that decision and thinking about scalability of concurrent users we decided to use a relational database system that could support efficiently 1 Million users. The tool chosen was PostgreSQL. However, in order to ensure the performance of the implementation of the models created to be merged in PostgreSQL with the console in Python we chose a powerful framework called Django REST. This Framework based in Python for building APIS’s was used to ensure secureness by managing authentication and management of users registered in the platform. Also a particular feature that we can use from Django REST was to serialize the information in JSON lists to handle requests with other languages and to not affect the database. This is a big milestone because it allows us to deploy in the mid-term the web app into a mobile app with stable programming languages such as Kotlin and Swift.
Finally, for the web infrastructure we analyzed that a LAMP stack despite to be reliable in the long term it won’t be enough to cover concurrent users and it will be in risk overload of the servers. Then we decided to take the leap and deploy our code base in containers as microservices. The containers will be managed and monitored by Kubernetes. Some suggested us to use Docker swarm but we researched and found out that Kubernetes not also will gave us more stability in the traffic and codebase but also a better performance with the integrated load balancer focused in the traffic based in nodes. By choosing this web infrastructure we also were aware that the deploy will be affected and mostly the MVP will be finished in production but the deployment could take longer than a regular LAMP stack. Nevertheless, this risk was evaluated and assumed in order to guarantee the performance of Commodoro platform for their users.
The next image will present our stack infrastructure based in microservices
The Development
In this point is where the blog takes a more nerd approach. Since the team has defined the idea and the scope of it, and considering the timeline to deliver a first MVP, we discussed and adjust the features and deliverables of Commodoro app. Therefore, the team focused in the implementation of a customized, professional, attractive tool, to create custom profiles based on the farm information typed in a form by the farmers. The experience has to be as simple as possible so we set up a 3 step interaction (farmer registry, filling information, profile created).
Backend stage
The first step was to develop API native with Django and Django Rest Framework (DRF). We selected a set of variables that will be needed to create a complete farm profile. We divided them in this way:
Farm Profile:
- Place: store data about the farm, its features, its description and some relevant info.
- Certificate: international certificates which the farm was awarded.
- Feature: coffee rating made by a taster, the table includes ten features per farm.
- History: information about the farm milestones.
- Auth_user: all about the user who creates the profile, the farm is linked to him.
- Product: characteristics of the main coffee offered by the farm.
- Picture: store the path to the picture that will be shown in the profile.
The relationship between the tables were defined in an Entity-Relationship model (ERM) as the following image
To design the ERM of Database for the MVP we chose these relationships to achieve a one-endpoint posting from the React server. A regular Django app is divided into separated apps, inside each app we can find the models that are represented in PostgreSQL tables. Those were divided by:
- Users app: User and Profile models
- Products app: Product and Feature models
- Places app: Farm, History, Certificate and Picture
Each one of these models needs a view function, or view for short, that is a Python function that takes a Web request and returns a Web response. Django Rest Framework has Generic Views, a type of Class View that makes it easier to code responses. This snipset contains the model of generic views.
class FarmsListView(generics.ListCreateAPIView):
queryset = Farm.objects.all()
serializer_class = FarmSerializer
class FarmsView(generics.RetrieveUpdateDestroyAPIView):
serializer_class = FarmSerializer
queryset = Farm.objects.all()With ListCreateAPIView we can handle GET and MOST methods, while with RetrieveUpdateDestroyAPIView DELETE, PUT and PATCH methods. URLs or Endpoints are linked to views, and views are also linked to a Serializer.
Serializers allow complex data such as querysets and model instances to be converted to native Python data types that can then be easily rendered into JSON, so the API can be consumed using the HTTP Methods. Inside Places directory we created a file named serializers.py where we defined the way as the data is represented in JSON format. Each model and view have a serializer, and the serializer can be nested, so it is possible to POST to all models using one view.
class ProductSerializer(serializers.ModelSerializer):
class Meta:
model = Product
fields = (
'id',
'farm',
'coffeeName',
'description',
'price',
'variety',
'processing',
'crop_year',
)
read_only_fields = ('farm', )We decided to POST all the information in the Farm view, due it is related to almost all models. The URL/endpoint for working is 127.0.0.1:8000/api/v1/farms/. We tested the API, as the following image shows.
With this the first part of the backend development is finished with the integration of a native API and the next step is to integrate the API with the frontend using ReactJS.
Frontend Stage
Now, for the frontend design implementation, we used some design patterns from ReactJS. React uses a very different paradigm from the traditional HTML, CSS and JS because it uses components, these components rendered, uses something called JSX, which is an extension of the JS syntax, it is a sort of mixture between html and JS. ReactJS have many advantages like the reusability of components and because it has a better performance when handling states, so when some of the components in the frontend change, it won’t affect the entire DOM.
In this stage of development, our main challenge was learning how to use ReactJS. This library is constantly changing, so even when the library itself had a lot of new concept for learn for program some block of code, at the same time there were other ways to do the same thing because of the constant improvement of the library i.e. We used hooks, promises and some React components, for some parts of the code, and these pieces of code not necessarily have to be correlated. It is normal to see this in some React projects because new features are added to the library.
The most complex view in Comodoro’s web page is the Farm Form page, which is the place where the farmers are able to create their profiles. It has some fields where the user put the information, also pictures. If the architecture of Commodoro were monolithic, input this information would be easier due the nature of a backend created for example in NodeJS would permit the information to be stored in the same layer of the code pattern. However, as explained before, we set a couple of objectives to make a solid platform so we saw in Python along with Django REST the right approach to handle the data from the users without compromise security and reliability of it. To accomplish this implementation basically the main component has a state where all attributes are stored, we created one key for each input space. It was designed so every field in the form must be filled, when this happen, the data is sent to the API in two steps.
- Images: we decided to send first the image for simplicity, when the user post successfully an image to the backend, our application will receive a response with the current image path in the API server, so other models can store this information.
- Farm Post: it is a complex form that includes different models, with a single post we could create all the necessary relations for a Farm in the database. All information was loaded in a state when the form was completely filled, then we make the post.
To support this task there were used AXIOS a tool that routed the information in JSON for other languages, but along with the API the information needed to have the same structure so there were no conflicts at the moment of store it in the database. After this action of POST, the information in the Farm form view, the user will be redirected to the new created profile with the corresponding information.
Web infrastructure stage
This stage is currently in development. As mentioned before, the team decided to handle cooperatively each main activity to get a viable MVP, therefore backend and frontend were the stages that took in development most of the five weeks to have the MVP. However, the team is still on track according the timeline and it has been containerizing the base code in two separated containers. In one container it was created the image of the backend base code, a second container is being implemented with the frontend code and a third container is being created to work as a database slave for new the data information of users.
It is important to recall that the Frontend container is going to have an extra configuration were a Nginx image is installed so it can be deployed to a domain already acquired https://commodoro.co. When the three containers are uploaded into Docker hub, the next step is to synchronize these containers with the google cloud platform and create the Kubernetes console in which the containers are going to be transformed into nodes according the Kubernetes paradigm. The advantage of implementing Kubernetes in this stage is that this technology allows a native load balancing of the nodes, that means that if the team wants can create a replica container of the frontend and backend and configure it in a way that if a traffic overload or unexpected shutdown of the page occurs either for a code that is being refactored or implemented, the replica will be activated and the page won’t be offline.
This feature that the team is aiming to achieve in the implementation is not easy to get, that is why the week 6 is being used for add this feature and ensure the reliability of Commodoro’s site due there will be a continuous integration and the idea is to made some test applying chaos engineering into the code itself.
The product
Here, is when all the process to get the MVP is completed. Commodoro achieved on time the creation of a platform that has a nice UX/UI and is aligned with the objectives proposed. The following video is the demo of the Commodoro web application and basically the coffee farmer is ready to create their farm as well as any user can see it.
We also have this landing page for the launching of Commodoro if you want to know more about the project and the journey behind of it.
The next steps
For Commodoro team we aim to extend the features of the web application are the following list we mentioned the TODOS as part of the phase 2 of Commodoro.
- Commodoro Cocoa, Avocado: One our main goals are to reach many farmers as possible so they can reach more customers and have a fair trade for what they produce. Therefore, we are going to add the feature of register farms from other commodities such as Cocoa and Avocado.
- Farm data visualization: We are going to take the data of the farmers to create and improve the visualization of their profile by adding more graphical data visualization of the performance of their farms. This will not only provide more accurate information to clients but also will provide insights of the farm performance for the farmers.
- Decentralized transactions ESCROW: We are going to implement a decentralized payment microservice based on ESCROW contracts so farmers and clients have the security of a transaction and delivery of the product. This microservice is going to be built in python
- Logistics tracking microservice: To support the scope of the payment microservice, Commodoro is going to implement an API of a Logistics company so the order can be tracked in real time as well as the status of the transaction
Conclusions
- In order to manage larger variables independently and heading for a future implementation of data analytics, we developed a nested multi-layer serialization in our API. Now, it’s easier to GET and the integration with components in ReactJS to achieve this, we had to understand complex relationship between models in our DB.
- We decided to use ReactJS for their components and capability to handle complex methods without sacrifice UI/UX. However, the paradigm of this library as well as the Hooks and Promises concepts were harder to understand with a python implementation.
- Implementing a POST request in REACTJS that can manage the multiple layer serialization in our API was a challenge that took 40% of the total project. In order to successfully post the data in the form to the API we had to use AXIOS and transform the input data into keys so the API can receive the data properly.
Finally, this last conclusion is not a technical one. We as team believed in the first moment of our member’s skills and take the leap to create something from nothing to design, program and deploy a full stack application. We encourage people who are passionate to also take a leap and think in a technological sustainable solution. Some may say that everything is already created and there is no need to do nothing more but there are unlimited opportunities in which your knowledge and passion will create something that can create a solution for a niche.
Thanks for read. Hope you liked this blog, stay tuned for more articles either for share knowledge or experiences.
