MALL Group is the largest e-commerce group in Central and Eastern Europe. It brings together more than two dozen e-shops selling a wide range of products from electronics to food.
The marketing department of mall.cz needed a tool that would allow them to quickly and according to current needs make changes to the website menu, for example for seasonal products, various events, etc. At the same time, it was necessary to ensure that the history of changes was saved and that it was possible to return to individual changes if necessary.
The main challenge was how to store the data so that a history of menu changes could be easily maintained: what edits were made by whom and when. Then also the promotion of the changes as such to the customer on the web.
We used event stream and snapshotting to save the data. In this case, the data is not saved as a whole, only the operations that are performed on the data are saved. By applying these events to the default state, we get to the resulting data.
The basic operations are creating/editing and deleting an item. The default state is an empty menu.
The important factor is how the operations come in sequence. One must always proceed from oldest to newest when applying operations.
Thanks to this system, we are able to compile an overview of how the data looked at each point in time. When we add user information to each record, we then have a straightforward history dump because we clearly know what operations were performed, at what time, and by whom.
But the problem arises when we need to apply many events at once. It is not good to always start from the initial state, because the application of all events will gradually increase in time. Snapshotting reduces this problem. This is the continuous saving of the resulting state of data at a specific time. If we have a state stored at a time, then we get to the next state by applying events from that time onwards. Suppose we store snapshots with every 100th event. We are then assured that we always get from the previous state to the current state after applying at most 100 events, and we are able to allow a constant time to build the current state of the data.
During implementation and use, some shortcomings of the chosen system emerged. However, its pros still far outweigh the cons.
The system is not suitable for the large amount of data that should be searched. The resulting snapshot and all operations with it are kept in memory. The system cannot use classical methods for searching in database systems. However, they are not needed in this system. This is because search is only required at the top level of the data and we were able to optimize this applicationally. Moreover, if the data is kept in memory, working with it is very fast.
Another shortcoming is bug fixing. In order to keep the data always constant over time and to be able to build it from the beginning to any point, we also need to maintain bug event handling in the code. Until the bug is removed, the data is still being developed with the bug. Since the bug is removed, the data is no longer evolving without the bug, but new event types must be created because of it.
We had to solve the problem of data distribution between instances, which happens asynchronously through the rabbitMQ messaging system.
We also protected the data so that the result is always stored statically in a file on HA online storage, so that there is a fallback solution in case the service is unavailable. This ensures that the menu always reaches the site.
We built a system that has a complete history, can easily go back to earlier versions of data, and quickly provides the data clients want.
One of the crisis areas of any web application is the speed of loading images. The goal of developers is to make sure that the user doesn't actually notice that any images are loading. If they do notice, it's not a good sign, and no one would be surprised if a user in such a situation quit the store for that reason.
As one of the largest e-commerce platforms in Europe, MALL needs to optimize the loading of the web application to make it as fast as possible. Of course, this is also related to fast loading of images. MALL as an e-commerce solution displays many different products in many different photos. We worked on a solution that appropriately optimizes virtually any input image into the smallest possible output compressed image. At the same time, we wanted to exploit the potential of next gen image formats like webp and deal with the fact that not all browsers use the format.
The biggest challenge within the project was to deal with several factors.
We had to cope with a large variability of input data. The input images are often machine uploaded and their quality is not guaranteed. In addition to having to deal with virtually any type of image, we had to be prepared for "broken" images, corrupted image headers, etc.
Mall has more than one million products in its assortment. Each of these products has many photos attached to it. We had to prepare a solution that can optimize up to millions of images per day. Since we used high compression methods and sometimes work with quite large images, one compression can take tens of seconds. So it was necessary to prepare architecturally for such a load.
For even better loading, we optimize ad hoc into several different qualities. Currently 3 different image qualities and two types of output are used - jpeg/png or webp. In the future, it is possible to achieve that mobile devices will primarily load smaller and lower quality images than desktop devices.
We opted for an AWS serverless solution because of the highly variable scaling of the service. We are thus able to scale the entire solution in a very short time and optimize millions of images.
The whole service works completely independent of any other service. Therefore, in order to be able to quickly scale the flow through a given service, we have prepared a REST API that works only to receive optimization requests. Its goal is to pass the optimization request to the queue and at the same time pass information to the requester about where to find the resulting image.
Then the actual optimization of the images takes place. We work with two variants. First, a very aggressive compression is performed, which can optimize the image to the marrow but still visually unchanged. However, in many cases (approximately under 1 percent of all images) the image fails to optimize. For these cases, we use a fallback option that can optimize (though not as well) virtually every image.
During the process, we had to deal with high resolution input images, but with one setting for the final optimization. In one of the solutions, we also counted on running multiple optimizations and then comparing them with the original image - i.e. to select the type of optimization that does not visually distort the resulting image in a machine-like way. However, after several measurement tests, it turned out that one method wins in most cases.
The resulting solution achieves savings of around 30 percent after the first deployment. After deploying the conversion to next gen formats, we got up to over 60 percent. We also measured the result of the whole event after deploying the first phase (without next gen format). We measured using the dom content loaded metric, where we improved both mobile and desktop by more than one second.
E-commerce customers may not realise that large e-shops are almost permanently improved by large teams of developers to make the shopping experience more and more user-friendly and simple. Attention should also be paid to such details as error reporting, which can discourage customers from making further purchases. If an error appears on the site, it should be reported as soon as possible so that it can be corrected quickly. We have been working on an app for MALL that can call up an error reporting form anywhere on the site. This form, once submitted, goes into the JIRA system and a request is filed here with all the details and information so that the error is specified as best as possible and can therefore be resolved as quickly as possible.
This is an application that allows:
Within the project, the biggest challenge was to cope with several factors.
The browser itself does not support a printscreen that can be sent immediately. We tried different libraries and finally used Html2canvas, which can rebuild the DOM in the background and make a printscreen for better orientation. The image is then cropped to the position and size of the browser window.
Due to the fact that the application is downloaded from another domain and integrated into a web page, the application cannot download images from the domain it is currently on (same-origin policy). Therefore, we created a proxy server to which the image link is sent and the image is returned to it.
The images are saved at the 3rd party so that JIRA does not accumulate images, and the image link along with the browser metrics will create a ticket in JIRA. This is also where authentication via oauth2 occurs.
It is a single script that initializes the header and starts the application when you click or press a keyboard shortcut. The application allows translations or can change settings via a global variable.
For the final solution, we used React on the frontend and a Node-JS server, which is a service that runs on AWS EC2 in multiple instances using AWS s3 and AWS SES to send email. The service could easily run on AWS serverless (Lambda).
There is an express on the server that intercepts the POST request from the frontend, it sends the message, image and browsing data, then converts the image to base64 so that the image can be stored as a string, and all the data is stored on AWS s3. Then it is sent to JIRA, where a ticket is created that has specific settings, e.g. it was created by a certain user based on an email.
In the event of an error, a ticket is created in JIRA that has all the parameters relevant to resolving the error set: you can look at the image and the browser metrics, and you can also see which user entered the ticket. The helpdesk can thus contact the user immediately or fix the error immediately.