Performance tips for production

All this is not mandatory for run MyDMAM and is just here for informations.

Always set up like you wants.

Services topology

By host/server/VM:

  • Two Cassandra nodes on two separated servers (one instance by server).
  • Two Elasticsearch nodes on two separated servers (one instance by server).
  • A MyDMAM path indexer (storage.yml > regular_indexing options).
  • A MyDMAM Web server (with the reverse proxy for HTTPS if you wan’t open this to worldwide).
  • A MyDMAM Metadata indexer/renderer
  • A MyDMAM Transcoder (for metadatas and/or for watchfolders)
  • A MyDMAM FTP server
  • A MyDMAM Watchfolders probe (2 for high availability)

For a lot of users (from 10 simultaneously or 100 sometimes), and/or a lot of files:

  • A big Elasticsearch cluster (see below).
  • Some powerfull servers for MyDMAM Transcoders (big CPU).
  • Multiple MyDMAM Web servers with a load-balancing web server in front.
  • A memcached server for MyDMAM Web server(s).

Networking

  • Isolate Cassandra cluster/ring in a dedicate network/VLAN.
  • Isolate Elasticsearch cluster in a dedicate network/VLAN.
  • Isolate Cassandra, Elasticsearch and MyDMAM clients connections in a dedicate network/VLAN.
  • Use entreprise grade and managable switchs. 1 GBE/server for db/web activities and 10 GBE/server for media access.
  • Avoid routing between Cassandra, Elasticsearch, MyDMAM and media access. Use only direct links with dedicated LAN interfaces.
  • Use LACP with 2 links to 2 separate clustered switchs for each servers, for even more stability.

Media and datas

  • Use shorted access to medias, avoid bridges like local mount with specific client, as you can.
  • For SAN access, try to setup instances on a SAN host.
  • For NAS mounts, try to setup instances on the NAS server, if it’s possible.
  • Use SMBv3… only if you can not do otherwise. MyDMAM can’t access directly to this protocol and actual drivers (Debian 8) are not very stable, with sometime an access lost and unmount.

Databases

  • A database server shoud never run other things as the database service, alone.
  • Use a maximum of RAM and CPU for these two services type.
    • 2CPU + 4GB RAM for small setups.
    • 8CPU (16 Threads) + 16 GB RAM for big setups.
    • (only 1 CPU + 1 GB for testing is enough).
  • Disable RAID for Cassandra servers, mount locally each hard-drive and declare each on Cassandra configuration. It will sparse datas for each drives. On each server on cluster.
  • Setup MyDMAM Keyspace on Cassandra with a replication_factor of 2. If you starts an clean setup, set it in MyDMAM cassandra.yml setup.
  • For big pathindexing / metadata storage (from ten millions of records) with Elasticsearch, thinks to setup a better sharding.
  • For a lot of users, thinks to setup separate storage-only/index-only Elasticsearch servers.
  • Use Separate partitions or disks for data/sstable files and temp/commit log. Both for Elasticsearch and Cassandra.
  • Don’t externalise Cassandra or Elasticsearch datas on a separate server or NAS. Use only local storage. Even if it is slower.

System

  • Use minimal OS setup and activated daemons.
  • Don’t mix MyDMAM/Db services and others things on same server.
  • Setup NTP and check if the time is really the same.
  • Install regulary security updates.
  • AppArmor/SELinux must be setup correctly if you needs to use it with MyDMAM.
  • Keep an eye of server logs and load.

Virtualization

With MyDMAM, absolutely all can to be done with virtualization.

  • Check NTP and servers time.
  • Separate the most of services as you can in VMs, and starting with one vCPU for each. Even the databases. One vCPU for small needs, two nodes for Cassandra and two nodes Elasticsearch work very well if your hypervisor’s CPU is strong.
  • Use enterprise-grade and validated equipements.
  • Setup VM Tools for each VMs.
  • Disable swap on VMs, enable swap in host for its.
  • Don’t start VM backups with all Cassandra nodes a the same time.

Don’t do, never

  • All setups in the same host (ok, but only for test).
  • Upgrade Cassandra or Elasticsearch savagely. Beware with the deb/rpm automatic upgrades. If you have installed them from repositories.
  • Forget to check NTP status for each server. I insist.
  • Change the default Java JRE to an other version or an other vendor before do test or understand the implications.
  • Setup, upgrade or recompile ffmpeg before test it with MyDMAM “by hand”. ffmpeg parameters can evolves with time and can to be conflicting with MyDMAM.
  • Shutdown all Cassandra nodes before shutdown all MyDMAM services. Anyway, after you will have to restart them by hand.
  • Mix some MyDMAM instances versions, above all with HTTP server instances. Serialization loves data format change.
  • Open a MyDMAM HTTP server access from worldwide without a SSL (TLS) reverse proxy in frontend.
  • Open something access from the worldwide to databases instances.
  • Have you checked in your servers the NTP service status ?