Making European culture accessible to everyone, everywhere

Author

Pavel Kats

CTO of Europeana

The Hague, The Netherlands

"BusyMachines brought a thorough understanding of technology to the table. This type of project requires distributed computing, where you develop a system that works on a big number of servers. They were up to the task, and we were able to report errors so these could be solved quickly."

Read full testimonial
  • Product

    Web crawler that aggregates the digital archives of museums.

    See Product Gallery
  • Value proposition

    Make European culture accessible to everyone.

  • Our delivery team — 3.0 FTE

    2.0Backend

    0.5PM

    0.5Techops

Delivery timeline

  • MVPDuration: 12 months

    Service: Dedicated Software Teams

    BusyMachines designed & implemented the distributed web crawler & data storage. We deployed the system on a cluster of 50 servers. During the first month after launch, our crawler collected & processed more than 50 million media items from 2000+ museums.

  • Post MVPDuration: 10+ months

    Service: Dedicated Software Teams

    BusyMachines provided continuous software development with a dedicated agile team. The main objective was to scale up of the crawler and reduce costs associated with the operational complexity and volume. We worked in short development cycles and released every two weeks.

  • Maintenance

    Service: Dedicated Software Teams

    Software maintenance & support delivered as part of the "Post MVP" development effort.

The vision of Europeana

Europeana works with thousands of European archives, libraries and museums to share cultural heritage for enjoyment, education and research purposes. Europeana Collections provides access to over 50 million digitised items – books, music, artworks and more – with sophisticated search and filter tools to help people find what they’re looking for. The dedicated thematic collections on art, fashion, music, photography and World War I contain galleries, blogs and exhibitions to inform and inspire.

The work of BusyMachines

The Europeana platform harvester searches and retrieves digital heritage media (ie. images, texts, videos etc.) from thousands of museums and cultural institutes across Europe. The value of the platform resides both in its access to unique content and the way it restructures and normalizes it into a true knowledge base.

  • BusyMachines is responsible for the development of the Europeana distributed data collector system. The challenges of such a system reside in its distributed nature and high efficiency requirements.

  • We also designed and automated the entire custom production infrastructure. The project codebase is open source and can be found on Github.

  • Technology stack

    BackendJava, Akka, MongoDB, Solr, Akka Cluster, Netty

    PMScrum / Agile

    TechopsDocker, Chef, Kibana / Grafana

  • Deliverables

    Cloud platform

    Monitoring web portal (B2B)

    Custom server deployment

Have a new project in mind or need help with an ongoing one?