Star Citizen - Ask Operations

JMasker.CIG

pd12 | pd12 said:
[hide]
[source]
Hi there,

I know you're working on overhauling the launcher. What features will you envision it to have? Would it be like Steam where it can automatically update only the files that are corrupted or need to be updated to latest versions through some hash checking or similar?

Also, are you overhauling with the Linux launcher in mind, and have you thought about using the Steam runtime for Linux?

Thanks! - for both your great work and great answers in the other thread. Real good insight into the DevOps (incl tools and processes and platforms [crossing my fingers for Australian GCE datacentre]) of Star Citizen! =D
pd12

Hey pd12,
Thank you asking our first question here! The new launcher is currently being developed to be as seamless from the website as possible. This means our launcher client will essentially be a wrapper for the website and current web functionality. We are doing this so we are not duplicating or re-inventing features players already have but in a launcher format. We are still working on improving the launcher's underlying patching technology; evaluating several different "off the shelf" technologies, as well as a possible in house bittorrent type tech.

Basic functionality of the launcher itself will be:
Login and Authentication, patching, launching the client, and modifying some client options (region, language, etc).

Benefit functionality will be:
Everything the website has (Chat, Organizations, hangar modifications, etc)

We are currently not looking at Steam at all. HOWEVER, everything is subject to change and the work we are doing now still needs to undergo an evaluation and further wire frame phase.

Hope that answers your question!

JMasker.CIG

Newman | Euthenia said:
[hide]
[source]
When do you plan on compressing the game's patches?

At the moment, it looks and sounds great, but the 20GB patches every few weeks are almost (almost) a bummer for those with slow internet as the game is unplayable for 1-2 whole days. I believe its re-downloading the whole game each time?

I love the work you all are doing. It's hard to express with words my excitement for this game. Keep up the great work!

Great question. Tough answer.
Right now we are not looking at doing patch compression. The reason is it is an added step of tech that we simply haven't had the time to fully research and add into our development pipeline. Also, compression is not a silver bullet and there are very serious drawbacks to compressing a patch, then un-compressing it, and then applying it.

There are several reasons why our game patches are so large right now:
1. We are delivering a lot of new content
2. Large numbers of file formats keep changing, which forces players to redownload all assets that have "changed"
3. Our current patching tech is terrible.
There were only about 11-20 ".pak" files, which are the CryEngine asset bundles. Our patcher cannot read inside of these bundles so if 1 file has changed in a .pak file the whole .pak file must be redownloaded. We have somewhat improved that problem recently when we made a change to use more .pak files, we are now at around 75 .pak files which makes the problem slightly better.

The real solution is to go to image dif patching, which is where the full game is on the CDN in a static image, the patcher difs your version of the game vs what is on the CDN and then only downloads the exact files that have changed. Even better if these downloads are using a technology that makes the most of your bandwidth and peer to peer filesharing; like bittorrent.

Needless to say, there is much more work to come.

JMasker.CIG

the_angry_angel | the_angry_angel said:
[hide]
[source]
Couple of questions;

1. Puppet, ansible, chef or are you some of those weird freaks that use cfengine?
2. Why GCE? Was it purely cost? Is there not a little bit of concern that since Google don't dogfood GCE, unlike Amazon (for example) that you'll get more painful problems to deal with in the long run?

I am going to try to answer a bunch of questions all at once.
We are using Chef for our config management.
And as for GCE, I answered that question some here:
https://forums.robertsspaceindustries.com/discussion/231526/around-the-verse-questions-for-jeremy-masker#latest

But I am not worried about the different problems we will run into on GCE vs AWS, since I have run into problems on both.

The company is working on the load times into the game and between the Hangar and Arena Commander maps. However, that is not an Operation team responsibility. We would need to talk to our engine programmers to understand why they want to keep the .pak files signed.

We are looking at more compression for our patches, but this is in a very early stage of research but LZMA compression looks interesting.

I would like the launcher to allow a user to edit basic game settings before entering the game, however this is not planned for Phase 1 of the new launcher, or Phase 2. So it would be many months before we will even evaluate how much work it will be to accomplish.

Our large Networking summit is in 2 weeks, so we still have not chosen a final Database yet. More on the results of that summit at the end of March.

I hope some of this is useful.
The Operations team will be on looking at these questions to answer in further detail later.

JMasker.CIG

GreatBeast666 | GreatBeast666 said:
[hide]
[source]
On the topic of changes to the launcher;

Will we see a 'current server status' feed, similar to the top of the RSI website -- within the launcher itself?

Ideally this is where it _should_ be, so whenever there's any sort of issue, those trying/unable to get into the game, at least know why - without having to open a browser and check the website header.

This is definitely a must have feature for the launcher. I agree, it is where it should be.

alex_dev_cig

deathstrike | deathstrike said:
[hide]
[source]
Elsewhere you've mentioned stuff about Google Kubernetes which I believe is a container management software. Are you guys using containers for the various game servers? I imagine having a control program spin up new containers on virtual servers to host more people when MP servers get full would be in line with what we've heard about how CIG uses google compute.

I've been working with Docker lately and it seems like it will be a boon to my field (HPC Research support) and I was curious if CIG is using Docker or some other type of container. And also if you felt they would be a boon to portability in the end because I know many people are looking forward to running their own MP servers.

We are indeed looking into containerizing our services, both in terms of the game servers themselves and the supporting services. We are using Docker in some areas already and are working with Google on the possibility of setting up our own Kubernetes cluster for testing. We like the portability that containers will afford us but also in terms of scaling at speed, there is currently no better option. Should this prove to be the best route, the beginnings of discussions about the possibility for people to host their own containers has occurred and while it presents certain challenges it is an idea we are considering.

Keegan-CIG

[TAC] Saii | Saii said:
[hide]
[source]
I would suggest to take a look how guildwars (2) handles and have handled game distribution for over a decade, technically near perfect if you ask me. Not only because they have had almost no downtime whatsoever (server does not have to go down for updates/upgrades.. they have over 99% uptime on 10+ years), but also by using one big packed and compressed asset file, updating and loading times are fairly quick.

So, to be a next gen game, having no downtime whatsoever should be a requirement. As we know it can be done.

This is something that's always on our radar. There's plenty of good examples of releases gone wrong. A couple games come to mind, such as Sim City, and we try to take these examples and learn from them.

That being said, we've not infallible, It's very clear we've had our share of rough releases. One of the biggest elements that separates us from traditional game developers is our constant release cycle. As a DevOps team, we're always prepping for the next release, staging the next PTU version of the game. These are our weekly drills, and as the game gets refined, so too does the release process.

To address uptime: we want to get to a place where we are able to migrate players from one server instance to another, in the event of a crash, or unexpected shutdown.

Keegan-CIG

Chris Longknife | Apoapsis said:
[hide]
[source]
I'm not sure if this has been asked, but what about a patching technology similar to what Blizzard uses where the game downloads the core files first so you can start playing and then finishes patching as you'e playing (not sure if this is practical bandwidth wise)? When I played World of Warcraft, it was always nice to only have to download a small part of the patch and then optionally I could instead of waiting for the full patch to finish, start playing while the rest downloaded in the background (with a small negative effect on game play / lag).

The DevOps team has been investigating multiple solutions regarding this. The ability to stream data will be a necessity, especially when we hit the point where the game spans multiple universes, many of which a player might not ever reach. There's no reason to download this data the player will never see or use, so why use the network bandwidth for it?

One thing I'd like to have eventually implemented is a connected data tree of the asset structure, with nodes connected based on their position in the world. This way as you move throughout the game, you traverse this data tree at the same time, and assets that you may need soon are pre loaded. But we'll cross that bridge when we get to it.

Keegan-CIG

Antan | Karmola said:
[hide]
[source]
Other than it being on the Internet, how is the Cloud different from the traditional idea of a Mainframe?

Utilizing cloud solutions for hosting game services is entirely a different beast. In a traditional model, servers are hosted on physical hardware, on server blades, so there are hard limitations on what is available to run servers on. In a cloud environment, it's much easier to get more processing power or memory. Scaling become a whole lot easier. As a consequence, the orchestration of servers becomes much less focused on "What hardware do we have to work with?" and more "What hardware do we want?"

alex_dev_cig

DiFarrami | DiFarrami said:
[hide]
[source]
Gentlemen,
in addition to @AdzAdama s brilliant post above https://forums.robertsspaceindustries.com/discussion/comment/4543157/#Comment_4543157 and your excellent thread and contributions, I would like to add some aspects to the issue of 'lag & loading':
All Oceanic customer base that is connected to the Asia data center will experience 230ms+ latency currently. There is a substantial amount of α-testers getting kicked from servers having to restart their clients far too often.
It is my understanding & hope that this could change once a Google Compute Engine service is active in Australia; But since this is a Google decision and not CIG however, this could happen whenever, or never.
Would Operations consider the following options :
1. Increase server kick latency threshold for the Asia server until an Autralian data centre is established ?
2. Consider a 3rd party cloud provider (like Amazon or Microsoft) as an Aust server to bridge until GCC is established ?
3. Include an 'inflate' option in the launcher to decompress PAK files locally (whilst still being able to have them PAKed and key locked if req'd) to decrease load times to est. approx. 1/5 of current?

Keep up your good work, and thanks for engaging with the SCcommunity ;-)

I will attempt to answer your and @AdzAdama’s question together. Global connectivity is something on DevOps’s radar separately from the server engineers who will be working on network optimization as an ongoing process. The parts we are looking at revolve more around the infrastructure and service availability near the end player than the parts that include the latency timeouts which are coded into services. As such, it is true that we are awaiting Google to make a decision in regards to GCE in the Australian region, however as you mention we have very little control over this and thus we are looking into some alternatives to implement if and as needed. Some ideas that have been discussed include finding a Point of Presence in Australia with stable and established backbone peering to GCE – one of the major causes of latency across distance is the number of BGP hops and the vagaries associated, if we could peer directly with a local provider and provide an end point closer to the user it would significantly improve connectivity. Also as we containerize our services it will allow us to be more platform agnostic, meaning we could possibly utilize other vendors or roll our own datacenter far easier and more cost effectively, increasing the chance of bringing services closer to players. We still have much to discuss and work on regarding these topics and nothing is set in stone but I will say that I think there will be improvement to the latency and ping times you are seeing currently as we move forward in development.

alex_dev_cig

DiFarrami | DiFarrami said:
[hide]
[source]
Thanks @alex_dev_cig for your quick response & clarification.
The proposed strategy will no doubt have greatly appreciated positive feedback from the oceanic community.
For part 1) could you point towards the right CIG team member / department that could look at this suggestion?
for part 3) is that something that you would consider, since the launcher is undergoing a redesign?

1) I will talk with the server engineers on this, but I think it is currently set to 45000ms which seems like it should handle this case but I will double check.
3) We are working on lowering the load times in general, I think Jeremy mentioned this in one of his posts but it is also more on the server engineers side as well. On the pak file side Keegen mentioned the concept of streaming which will minimize some of the load times as well, also @Sam Redstone's comment above is something we have been discussing in regards to in-game location/navigation and the effect it has on paks that are loaded. We have not specifically discussed "inflating" as you mention but I'm sure we'll discuss it now that it has been suggested.

Thanks for the questions.

alex_dev_cig

Loc-Deu | Loc-Deu said:
[hide]
[source]

alex_dev_cig | aperuyera_CIG said:
[hide]
[source]

DiFarrami | DiFarrami said:
[hide]
[source]
Thanks @alex_dev_cig for your quick response & clarification.
The proposed strategy will no doubt have greatly appreciated positive feedback from the oceanic community.
For part 1) could you point towards the right CIG team member / department that could look at this suggestion?
for part 3) is that something that you would consider, since the launcher is undergoing a redesign?
1) I will talk with the server engineers on this, but I think it is currently set to 45000ms which seems like it should handle this case but I will double check.
3) We are working on lowering the load times in general, I think Jeremy mentioned this in one of his posts but it is also more on the server engineers side as well. On the pak file side Keegen mentioned the concept of streaming which will minimize some of the load times as well, also @Sam Redstone's comment above is something we have been discussing in regards to in-game location/navigation and the effect it has on paks that are loaded. We have not specifically discussed "inflating" as you mention but I'm sure we'll discuss it now that it has been suggested.

Thanks for the questions.
Hello, what did the Server engeneers answered? ( 1. Question ) Would ne nice if such questions would answered after such infos...

There are various services and they have various timeouts, the initial connection does have a 45,000ms timeout, but some of the other services have much shorter timeouts, Matchmaking for example. So at the moment it is possible that in certain scenarios you are exceeding the timeouts from Australia. In the end all these values need to be examined and balanced by the various teams and we are working towards that.

JMasker.CIG

[TAC] Knights-Errant | Knight02 said:
[hide]
[source]
I have no idea where to post this one.

I do not have enough disk space to DL the PTU for 1.0. I suspect that I will not be able to DL the 1.0 release. My current DL size (which includes 3 previous PTU DLs) is 54GB. I have less than 5GBs of space left on my SSD. I have 1TB of onboard storage with fast write time and an external hard drive, am not sure if there are any good work-arounds that will allow me to run SC from those locations (naturally, onboard is preferred). I take it that, If I wish to keep playing the public releases, I will have to purchase a larger hard drive. Is that true?

I understand the optimized release will be smaller, but if I am going to need a larger SSD, I may as well get it now instead of later since 58GBs (on my total capacity 128GB SSD) is about all I can muster on what is already a streamlined system (and otherwise awesome CPU)

Please advise soon.

System Information: i7, Win8.1, Nvidia Geforce GTX980, 128GB SSD with storage...it's the Alienware Area 51 R2 base model with upgraded Nvidia card.

Please advise freely, I was planning on upgrading it as the full release approached anyway so budget money has been collecting for some months now...just had not made purchases. Seriously, advise freely. :-)

Hey Knight, I hate to ever suggest to a customer to purchase something; especially something they may not need.
So, first I would suggest to you to clean up space on your SSD so you have more free space for Star Citizen. If you really can not free up any additional space, then I would research folder linking and see if you can move the PTU to your mechanical harddrive. If that becomes too complicated, then yes, I would suggest that you buy a larger SSD (assuming you want to keep the performance level you currently have).

Star Citizen is only going to get bigger. There are millions of art assets yet to be created, and eventually they will all need to live on the players harddrive. I am not sure how much more "efficiently" we will be able to package the game to reduce overall size. My instinct tells me any savings we get on cleanup and compression will be quickly undone by the shear number of assets we are generating and placing in the game.

Hope that helps!

JMasker.CIG

Booyaah | Booyaah82 said:
[hide]
[source]
Mr Masker,
JMasker.CIG | JMasker_CIG said:
[hide]
[source]

Basic functionality of the launcher itself will be:
Login and Authentication, patching, launching the client, and modifying some client options (region, language, etc).
Can you guys consider moving login/authentication from the launcher to the actual game client? Basically I would like to be able to switch accounts without having to leave the game (immersion breaking! :D). This would be similar to say how wow does it.

Here is what I was thinking would be a good hybrid solution. You could keep current login stuff on the launcher and have a checkbox below called 'Auto-Login'.

So if you fire up the StarCitizen.exe file and not the launcher, it would:
1) Do a version check to see if the local client matches the latest build and display an error message if the local game client is out of date and halt loading anything if so. Maybe ask them if they want to load the launcher and then exit the game.
2) Assuming the builds match, load opening credits splash (I hope you guys will offer a command line argument to -skipintro like Arma does please)
2) Show an in-game login splash screen where the user can login/authenticate inside the game client
3) User logs in and goes to hangar or character screen in the future

Now if you open from the launcher:
1) Login first
2) Do version check (please add a prompt that says 'new patch available would you like to download now'. I hate that it starts auto patching without asking me for consent first (i.e. I got stuff in my USER folder I might want to backup first)
3) Patch if necessary
4) If auto-login button is checked the game client will load, but will skip the in-game login screen and go straight to hangar or character screen

What do you think?

Hey Booyaah, while this idea certainly isn't out of the realm of possibilities, we already have enough work cut out for us already just to build a basic launcher/patcher that works. I only have 2 engineers who are working on the launcher, and they are under tight deadlines. Right now, we are not planning on supporting multiple account inside of login.

That could be something we work on for later phases of the launcher.
Thanks!

JMasker.CIG

LiquidAurum | LiquidAurum said:
[hide]
[source]
How do you handle your backups? Do you send them offsite? If so how is it automated? Via Software? Scripts?

Liquid, we have no backups right now, because we have no game database right now. Obviously, this will quickly be changing and be something we must think of.
How the Turbulent team does backups for the website DB is a good question. I will ask and figure that out.
Thank you!

alex_dev_cig

Schweinepriester | Schweinepriester said:
[hide]
[source]

JMasker.CIG | JMasker_CIG said:
[hide]
[source]

LiquidAurum | LiquidAurum said:
[hide]
[source]
How do you handle your backups? Do you send them offsite? If so how is it automated? Via Software? Scripts?
Liquid, we have no backups right now, because we have no game database right now. Obviously, this will quickly be changing and be something we must think of.
How the Turbulent team does backups for the website DB is a good question. I will ask and figure that out.
Thank you!
i'm sure there must be some context to the question from @LiquidAurum but the source-link doesnt work?
because it does sound like you have no backups at all!?

i very much hope you do have backups of a lot of stuff, e.g. source code and assets!
if not, that would the highest concern the community should raise...

A quick clarification on what Jeremy said about not having backups, we are focused on game live/dev ops and not on IT for the company as a whole, IT does have multi-site SAN replicated storage and off site backups for things like art assets, code and reference materials. We do not manage those systems directly.

alex_dev_cig

Booyaah | Booyaah82 said:
[hide]
[source]
It sounds like they have glossed over the concept of using a database to store game content for a while and have instead just been using the OOTB CryEngine XML files to store game data. I'm guessing CR didn't hire any kind of DBA back in 2012-2013, and now they are having this oops moment. Man is it really strange to think that no one thought about using a database earlier, or naive to think they were going to get away without using one earlier.

This will probably be a major development setback IMO, it will take time to develop a proper schema, procedures, reporting, backups, etc. Just make sure whatever DB stack you ultimately choose, that you can easily find a DBA or two and a few developers to build everything out. It sounds like they have been entertaining a few different options, my advice would be to not trust too much these small mom and pop shops, their marketing ppl will promise you the moon, and will try to sell you using a bunch of meaningless techno babble. I've had clients I work for go against my advice when I thought a 3rd party product was bad, yet they thought the product just sounded too good, decided to go with it anyways, then got burnt when the product didn't deliver.

One thing I might suggest is, if you are not that familiar with current database technologies, is to just pay a small premium and bring in an outside 3rd party consultant temporarily who does not have an agenda or slant with any of the DB vendors who can properly give you some advice. You'll proper want some type of NoSQL solution for your unstructured data (search, etc) and a more traditional RDBMS for reporting and storing core data entities.

Booyaah,
The data model for the game has undergone various changes since the inception of Star Citizen, the fact that we are building it in a modular way has led specific parts to have different solutions. The game engineers and we in Devops have experience with multiple types of databases and are actively working on determining what the best model will be. We are also consulting with 3rd parties as you suggest for new ideas and processes. Most likely we will end up with multiple solutions as the different parts have differing requirements. For example we use a traditional RDBMS for the web platform including the store, however this might not be the best thing for the economy server or the PU database. We are looking at many NoSQL products, also looking into Google’s datastore as we are on their network already. We are actively looking for a Data Architect to spearhead this development effort but until then Devops and Server engineers are working on it from our respective sides. We are definitely aware of the importance data will play especially in the persistent universe and are not letting the design of it fall by the way side. As Jeremy mentioned we have a big summit coming up where many of these topics will be discussed.

JMasker.CIG

Isogen | Isogen said:
[hide]
[source]

Trip Rodriguez | Trip said:
[hide]
[source]

JMasker.CIG | JMasker_CIG said:
[hide]
[source]

Star Citizen is only going to get bigger.
But...... the description of the Star Citizen "Spaceship shaped 16 GB USB drive" said we could put our Star Citizen install on it. That means the finished client has to be small enough to fit doesn't it?

[snip]

@Trip: Ben has mentioned the size of the USB drive will likely change to something like a 32G or larger depending on the game and cost to make it.

Hey Isogen, the USB drive can still happen, it would just contain a subset of the game. Like a starter package.
We are not a normal PC game with a "finished" data set that is Gold Master'ed and put on a DVD or Blu-Ray. So anything that is a download from physical media will likely only be a partial package that will need updating from the launcher.

JMasker.CIG

Michael Wells | Korren said:
[hide]
[source]

LaserVisor:
[hide]
[source]
rather than attempt to download 100 GB (or however large the game will be by then) and destroy my monthly limit twice fold.
1) The game files will be significantly more compressed and optimised than they are now. The size of the final game would probably be more like 30-40GB. (But this is still a very rough estimate.)
2) Game patching will also be more optimised, so minor patches would be in the hundreds of MB, and content patches would be a few GB.

As I have already said, I would not count on this. The game compression and asset removal is unlikely to yield such high gains that we will be able to reduce our client size to 30-40gb. The size and number of assets that are left to deliver means that our client size is much more likely to be 100gb.
Also, yes we are optimizing game patching for speed and to only deliver diffs, but this is unlikely to reduce actual patch size. Again, each patch has 100s of assets, each of these assets are at times 200mb, this leads to 2-6gb patches, and if we end up doing a file type re-factor and have to re-download 30-40% of the assets on the hard-drive, then the patch will be 14-20gb.

JMasker.CIG

Nick | superflytwo said:
[hide]
[source]
If a player encounters a large fleet of say 10 ships, will variation in spacecraft affect client side more then 10 of the same ship? Meaning if you run into a hornets nest of 10 hornets will the clients computer load that better since its 10 copies of the same data more or less the 10 mixed ships all with different data?

Great question. The answer is, I do not know.
Conceptually, I think the answer should be yes, since you are only loading one ship model.
Still, there are a ton of variables here: Paint jobs, weapon load outs, damage states, etc.

Also, though you might load those 10 ships faster, it wouldn't mean that your client will process that data any faster.
Obviously, if it is 10 Retaliators vs 10 Auroras the load on the client will be vastly different in those cases.

We will need to do more testing internally to come up with a definitive answer. However, assuming you have a computer that can run Star Citizen, in the scenario you outlined, I do not think you will realistically be able to note a difference between loading 10 Auroras vs 10 different ships.

JMasker.CIG

-*_GunnyM61_*- | GunnyM61 said:
[hide]
[source]
This sounds like a stupid question but I cant find anywhere that tells me what version I have. I thought I downladed version 1.0.2 but not sure if I downloaded 1.0.3 can you tell me where to find this so I know whether or not I need to download the 1.0.3 version ? Where do I find the 1.0.3 to download if need be ?

Thanks

If you open your Star Citizen directory and go to StarCitizen/CitizenClient/Client_Release.id and open it with notepad, you will be able to tell what your release version is that you have downloaded.

JMasker.CIG

Auric | AuricSilverwing said:
[hide]
[source]
Question about the mammoth projected download size: Does it all *have* to come from CIG servers?

There's a good number of South Africans that would want to play the game, and a lot of them will be able to pull large files from local (read: SA) servers without incurring large costs or waiting times. Will there be an option for someone to host a local mirror of the final client?

Hey Auric,
The answer to your question is both no, and yes. No we don't want you to have to get all your data from us, but yes that is currently how it is implemented. We push the game and patch data to our CDN (Amazon) and players using the launcher hit the CDN node closest to them to download the data. So if Amazon has a node in South Africa then your speeds will be much higher. Unfortunately, it looks like the closest edge node for SA is either Europe or India. All your data must come from the CDN for you to play.

However, this is today's solution. We are in the process of drastically overhauling and rewriting our launcher and patching systems. Our goal is to reduce patch sizes through better diffing and hopefully improve speed at location delivery. We will be writing more specifics about this system as it gets closer to fruition.

JMasker.CIG

Wing Commander MT | MTMnet said:
[hide]
[source]
for EU...

Amazon node Ireland and Germany (Frankfurt)

since...23 Oct. 2014
Germany Frankfurt
http://aws.amazon.com/de/region-frankfurt/
http://aws.amazon.com/de/about-aws/whats-new/2015/01/19/amazon-elasticache-is-now-available-in-aws-eu-frankfurt-region/

for Germany..
...what is used if I select "Europe" in my launcher... ?
should be the Amazon Frankfurt node ?... right.?

Selecting Europe in your launcher sends you to the Arena Commander servers hosted in Google's EU region. However, it also only match makes you with players who have selected Europe.

JMasker.CIG

Mike Edwards | Mike_Edwards said:
[hide]
[source]

JMasker.CIG | JMasker_CIG said:
[hide]
[source]

Wing Commander MT | MTMnet said:
[hide]
[source]
for EU...

Amazon node Ireland and Germany (Frankfurt)

since...23 Oct. 2014
Germany Frankfurt
http://aws.amazon.com/de/region-frankfurt/
http://aws.amazon.com/de/about-aws/whats-new/2015/01/19/amazon-elasticache-is-now-available-in-aws-eu-frankfurt-region/

for Germany..
...what is used if I select "Europe" in my launcher... ?
should be the Amazon Frankfurt node ?... right.?
Selecting Europe in your launcher sends you to the Arena Commander servers hosted in Google's EU region. However, it also only match makes you with players who have selected Europe.
Will it be also like this once the PU goes live? I have a lot of Friends in North America and it would suck if I couldn't play together with them.

No, this is not the end goal of the infrastructure. Very soon we will remove the separate game environments (NA vs EU) and combine them into one. At that point players will log into the game and depending on geoIP and the last log out location they will be routed to a GCE server that is closest to them. Network traffic between game servers will be run over the Google fiber backbone. We have yet to run tests to determine what that latency will be, but it should be much faster than routing this traffic over the internet.

You should be able to play with any of your friends from anywhere in the world in the game.

alex_dev_cig

I will try to speak to the global speed/location/latency questions posed by Bunny Harlan, Andrewzz, Aesop, Rocker, BoMbY, Oberscht and Napalm. We are currently during this phase of development on GCE only, homed mainly in the US Central zone, though we do have nodes in the Europe West region as well. Currently as we are preparing for the 1.1.0 launch 1.0.3 is on US Central only which explains why in your traceroute you are seeing the hops reach the U.S. This should change when we release 1.1.0.

In regards to global play latency, as I mentioned before we have not thrown out the idea of setting up systems on other providers or even building out our own datacenters to bring the initial connection point closer to the clients (so in Andrewzz table the Sao Paulo EC2 node is much better latency), and then using dedicated carrier bandwidth on the backend to reach whatever server might be needed. Based on some tests we have done, this could cut latency by up to 60% to the end client.

There are two aspects talked about here, there is the CDN download of patches (game) which is currently on AWS and this change was just made in the last patch to 1.1.0 on the PTU and thus not everyone may have seen these speed improvements yet. A CDN by its nature will try to connect you to the closet node to you to ensure the best speeds, we are trying to use this model for our gameservers as well which is what I explained in the paragraph above. Amazon uses its own geoIP look up and our platform is using Maxmind.

We are still evaluating a number of solutions to these issues but we are committed, as we have been from the beginning, to this being a universe that players from all parts of the world could play in together and none have to sacrifice gameplay quality to do so. Some of the optimizations are coming from the engine and server programmers and others from us here in the devops team, plus the partners we have are offering some wonderful solutions as well. I think as we move through this year and start introducing more multi-player aspects you will begin to see some of these optimizations in action.

JHolley.CIG

Rank Badjin | Rank_Badjin said:
[hide]
[source]
Finally after over ten years, CCP released two factor authentication for EVE Online. Is it too much to hope that RSI/CIG can implement this before launch?

Rank Badjin
(Paranoid Security Geek in RL)

Two-factor authentication is definitely a priority we hope to have addressed before the launch of the game. That's primarily being handled by the team responsible for the website (which is a different team than DevOps). I know they are currently working on it.

Keegan-CIG

pd12 | pd12 said:
[hide]
[source]

JMasker.CIG | JMasker_CIG said:
[hide]
[source]

Both of these are due to bad patch data on the CDN.
We have deleted the bad data, and regenerated the patch manifest that points players to the CDN's data.
This is not the first (nor the second) time errors from patching occured from bad patch data on the CDN.
Could you please identify at which point the data becomes corrupt in the deployment process and explain how and why that happens? Thanks!

The current patch build and deployment process is very needy and manual at its current state. We are undergoing a large overhaul to both patching and the build system, and in the interim are trying to maintain what's in place until it's time to do an Indiana Jones pedestal swap.

Any of the following can result in a broken patch in the current system:
1. Human Errors from Patch Creation Tool Use. This is anything from:
Selecting the wrong patchTo/patchFrom versions for a patch
Not adding every payload of the patches to the patch manifest files
Skipping over the build process for any of the dozen patch payloads we have
Incorrectly configuring any of the other around ~120 different patch settings

2. Patch Upload Process
Old internal upload scripts failing on newly implemented patch features
Manual upload tools failing during an upload, or only uploading partial data

3. Deployment flagging process
Incorrect manifests being uploaded
Old patch link paths on CDN not being invalidated when overwritten by new patches

And much more. There's a long list, but we're working to replace the more fragile parts of this process, and to automate the segments that are prone to human error. The new build and deployment systems are DevOps number one priority right now.

Keegan-CIG

Isogen | Isogen said:
[hide]
[source]
Hey team,

So with 1.1 you guys did some improvements to the patching system and launcher. Can you give some info on what's been changed? Also, will we ever get the P2P patching system?

Currently I'm heading development of the backend of a prototype patcher, that is intended for deployment to PTU in the future. It is a torrent based patching system. (which if you're clever and put the pieces together will realize that it was no coincidence that a .torrent file of our game was among the files leaked). We're still juggling around the different ways to implement the torrent delivery, but we've been doing initial internal tests of the launcher to good results. Fresh downloads of the game are quick and painless, and the patch creation process is far simpler. Internal users were experiencing speeds on average of 15 MB/s, which we hope to improve, and some were topping at speeds of 50 MB/s.

Incremental patching is still an optimization question. We need to take a look at how our pak data is organized, and structure it in such a way that complements torrent's internal mechanisms. Once we've gotten to that point, we can make an honest and fair evaluation of this compared to our current patching system.

P2P brings some legal concerns. Other game companies have had issues with lawsuits involving P2P. There is a current patent out on this method of file distribution. So instead of using p2p with torrent, we plan on using our Amazon CDN as webseeds.

-Keegan

alex_dev_cig

obscurehero [LF] | obscurehero said:
[hide]
[source]

VikMorroHun | VikMorroHun said:
[hide]
[source]
"P2P brings some legal concerns. Other game companies have had issues with lawsuits involving P2P."

Out of curiosity:
What would happen if CIG modified the User Agreement clearly stating the game uses P2P technology and by agreeing to install the game the user automatically agrees to that and also forsake (if that's the right word) any legal claim involving P2P against CIG?
Was talking about this in chat and finally, am posting and was posted about already.

Other games use libtorrent within their launcher to distribute content, can someone explain what the current legal concerns are? @Toast_CIG

The use of libtorrent itself is not a legal concern, the concern is specific patents which reference the use of P2P networks for the delivery of content - companies that utilize torrents can use webseeds as we plan to or they could license the technology from the patent holder. Licensing, at the moment, is not cost effective as the patent holders sometimes demand exorbitant fees for such usage. However, we are keeping the option open for the future.

Alex

Thexder1

Icematt12 | Icematt12 said:
[hide]
[source]
My download speed for Star Citizen is decent, very close to what I get on Steam, but my issue is the size since it takes a large part of the day.

Do you see increasing the number of .pak files used help in any way? Would that be a pain to setup from the current ~12?

We have been investigating many different ways to reduce the size of the patches, the most commonly discussed is the increase the number of pak files which should help reduce the size of the patches. We will likely start with increasing the number of pak files to help reduce patch sizes then go from there to see if we can come up with more efficient ways of handling this.

JMasker.CIG

Maltavius | Maltavius said:
[hide]
[source]
Delta patches?

WonkoTheSane | jaryiel said:
[hide]
[source]
Maybe investigate "rsync"?

On patching, please remember it is not quite as easy as just using rsync or "delta patching".
If you simply have a few files, or a single exe you need to update, these methods are stellar.
When you start trying to do a delta of a signed, compressed, encrypted PAK file, with 1000s of files inside it becomes a bit more complicated. Normal diffing programs can not see inside the PAKs to determine what changed, they simply see that the 1 file (the PAK file) changed, so they force a download. (Even that is an oversimplification, its actually more complicated than even that)

Again, not impossible to solve, and we are working on several ways of changing which files are put in PAKs and how they are arranged inside of PAKs. Some of these changes require changes to the game engine, which are not as quick as simply changing how the build server asset job builds PAKs.

RSI Community Forums

Howdy, Stranger!

Categories

Star Citizen - Ask Operations