by Perforce
Helix Core is a foundation for game development. It provides a version control repository for all digital assets — a “single source of truth” — from source code to art and other multimedia and released binary assets. Small teams with less than ten users do their work on Helix Core. So do thousands of users working on a single repository, which may include hundreds of TB or even PB of data.
It also has great support for distributed development, supporting teams spread out in different offices or countries. Proxies and replication technology give users the performance of local working. At the same time, they can coordinate their work with colleagues remotely in a seamless way.
With the change in working driven by lockdowns around the world due to COVID-19, these features are even more important. Previously, it may have been remote teams in other offices. Now, it is remote individuals.
But it is still vital to be able to:
- Coordinate your work.
- Sync many GB of files.
- Make it easy to lock files that you are changing and have your colleagues see that lock status immediately.
Smaller games companies have long embraced the cloud, setting up their Helix Core servers and accessing them remotely. Amazon Cloud Formation Templates and Azure Resource Manager Quickstart Templates for Helix Core now make this even easier, as well as accompanying best practice guides.
In addition, game development studios with one or more offices often find it useful to work with servers in the cloud. This is often as part of a hybrid topology that includes one or more on-premise central Helix Core servers.
In this blog, we will cover some of the options with Helix Core to support your games dev colleagues from small to large installations.
How to Use Helix Core Remotely
Here’s how to use Helix Core to support remote work.
Use a Single Helix Core Server
Helix Core is easy to install and doesn’t take huge amounts of resources to support teams of various sizes. Obviously, you need to consider basics, such as CPU, RAM, and storage sizes. But we still come across customers where a consumer grade workstation has been happily supporting the company for years! While a majority of installations run on Linux, there are plenty of Windows shops working happily, including customers with thousands of users.
Typically, we recommend higher speed CPU cores (rather than just more of them), and sufficient RAM to cache all the important metadata database files. Use SSDs for storage of metadata and log/journal files, but depot (or archive) files can be stored quite happily on (cheaper) NFS storage.
Equally, it is important to ensure you have appropriate backup mechanisms in place and are following recommended processes for regular checkpointing and verification of your archives.
File Types and Compression
Make sure that your system is correctly configured so that appropriate files of different types are compressed appropriately. For example, some files compress very well, while as others such as many image formats and Office documents are already compressed. For these types ensure the Helix Core filetype does not attempt to further compress the document as this uses CPU for little benefit.
Admins can configure the defaults globally for all new files so that most users don’t have to worry about it.
In addition, remote users should test the workspace compression option to see if it is beneficial for syncing lots of files.
Parallel Syncing
This can reduce the time for a sync operation of many (large) files significantly. Admins can set this by default, and when it is enabled, users can configure their own settings.
Cloud Instances
Installing in the cloud is straightforward. But it’s sometimes complicated by the ever-growing array of possible instance types and other options to choose from!
On AWS you need to be aware of the networking constraints which come with your instance type. We have had customers complain about performance only to discover that they were choosing a low-end instance and the network was being throttled!
For example, (at the time of writing) a T2 basic instance is “low to moderate” network, a T3 is “up to 5Gbps”. We would typically recommend starting with at least a c5n.large — while only 2 vCPUs and 5GB RAM, the networking is “up to 25Gbps”. Azure instance types have similar networking limits to be considered.
It is very easy to change the instance type if you are hitting performance limits. This requires only that the instance is shutdown, the type changed, and then restarted.
Choose Storage
On AWS we recommend EBS for metadata and log/journal. Many customers use EBS for depot files, but multi-TB filesystems start becoming more expensive. EFS is an option for depot files which is cheaper than EBS and infinitely expandable. While we have seen throttling issues in the past, AWS has been working on addressing these.
For Azure, Premium SSD is recommended for metadata/log/journal, and Standard storage can be used for depot files.
Monitor Performance
It is important to monitor the performance of your instances — for example, to check if the instance type needs to be changed or similar. Standard AWS/Azure monitoring dashboards are very capable. And it’s easy to install P4Prometheus together with Prometheus/Grafana to have real time monitoring of the Helix Core server itself.
Backup
This is easily addressed with AWS Data Lifecycle Manager or Azure Recovery Services. Just make sure you do it!
Manage Other Cloud Concerns
There are other cloud concerns you’ll want to address as well.
Data Ingress/Egress Costs
Don’t forget in your planning to at least estimate how much data you are likely to be checking in during development of your game. Also, how much data will individual users use on a regular basis in their local workspaces. In typical usage, 95-98% of the time is spent reading from the repository vs the small amount of time spent submitting changes or otherwise writing to it.
Security
This is a big topic and needs to be appropriately addressed. It’s important to consider proper management of roles, access, ports being opened, and ideally to whitelist IPs rather than have your server open to the internet. It is standard to have Helix Core server running over SSL to ensure traffic is encrypted. Encryption of storage volumes is also possible. Requiring users to sign in to a VPN before having access is also a recommended option.
It is straight forward to connect your server to Active Directory or other federated IdP (Identity Provider) offering multi-factor authentation. This includes third parties such as Okta as well as the cloud vendors’ own solutions. This makes user management easier, and can also implement SSO (single sign-on).
Leverage Helix Core Replication
As previously mentioned, you can set up servers around the globe that allow remote users to work as if they are local. This can be done entirely in the cloud, entirely on-premise, or in a hybrid model.
Using the cloud allows you to take advantage of their network backbone which can be a major performance enhancer. It can also help to reduce data ingress/egress costs significantly.
Replica Types
These include:
- Proxy – simple caching of file contents. These are easy to install and manage, and can be very effective with the larger digital assets common in games development.
- Forwarding Replica – the replica also has a local copy of repository metadata which means many query operations are handled locally. Write operations (which includes updating workspace state) are forwarded to the central server. These are often known as “smart proxies”.
- Edge servers – these support all actions locally with the main exception of submitting changes. The workspace state for all users using the edge is managed by the edge server itself providing local performance. You should consider how the edge server is going to be backed up (unless using a “build edge” as described below).
A common topology for multi-location sites is:
- Smaller offices have a proxy shared by all users in that office.
- Larger offices have an edge server.
- A forwarding replica is used for high availability/disaster recovery.
Personal Proxies for Remote Working
Individual users can have personal proxies. This means running a proxy process on your local PC. When you do a sync, the proxy digs the files out of the local cache. This is much faster than going up to the server and transmitting it over the wire.
In addition, you can schedule tasks to pre-populate the proxy cache with the latest builds. And you can set it to run in the early hours of the morning before you start work so that first sync of the day runs quickly.
Local proxies are easy to manage, and can be surprisingly effective, especially if you are switching between related streams.
Support Build Farms
Use an edge server near to any build farm to support local syncing by the build farm clients, known as a “build edge”. It makes the performance so much better: the edge server receives constant real-time updates from its upstream commit server and provides local working performance for the build farm. And you can use the cloud to optimize utilization and backbone communication.
A “build edge” may not need to be backed up, since it can be reseeded from the master relatively quickly, and build farms will just recreate their workspaces.
Replica Chaining
More advanced topologies include chaining of replicas. This can reduce the transmission of duplicate content between servers which may be useful for working between continents, e.g. Europe and APAC or the USA.
Remote Work Is Better With Helix Core
With Helix Core, it’s easier for teams to work remotely. If you’re interested in getting started, you can get a free trial for up to 5 users today.
Navigating Now and Beyond
This is part of our Navigating Now and Beyond in partnership with Sports Interactive series - guest articles and guides providing technical and practical advice for games companies to navigate working now and beyond the current Covid-19 crisis.
Find more articles at our Navigating Now and Beyond hub here.