Server to Server Data Transfer Technology

[AshuraSg]

Hi Guys, I hope to gather some ideas or recommendation on a certain scenario that I am facing right now.

Currently I have database server that is kept in a different room from my main data center. This is because this database server is meant for the staffs to do ad hoc uploading of data that are to be process in another server that is currently in the same room as of the database server and at the same time restrict them from entering the main data center.

But we are having the intention of moving the database and processing server into the main data center but still retain the ability of allowing the staff to carry out their ad hoc data loading in the previous room.

Therefore we would like to see if there is any technology to carry out such configuration at an optimal speed.

Things to note is that the database and processing servers are not connected to the intranet of the firm but is of a LAN of its own. Secondly we understand that one of the way is to have a direct network connection from the data center to the room but we would like to see if there is any other method to do this instead.

Thanks in advance.

 

[davidktw]

Hi Guys, I hope to gather some ideas or recommendation on a certain scenario that I am facing right now.

Currently I have database server that is kept in a different room from my main data center. This is because this database server is meant for the staffs to do ad hoc uploading of data that are to be process in another server that is currently in the same room as of the database server and at the same time restrict them from entering the main data center.

But we are having the intention of moving the database and processing server into the main data center but still retain the ability of allowing the staff to carry out their ad hoc data loading in the previous room.

Therefore we would like to see if there is any technology to carry out such configuration at an optimal speed.

Things to note is that the database and processing servers are not connected to the intranet of the firm but is of a LAN of its own. Secondly we understand that one of the way is to have a direct network connection from the data center to the room but we would like to see if there is any other method to do this instead.

Thanks in advance.

If there is no network connectivity from source to target, then you can forget about any realtime or sub realtime replication or synchronisation techniques.

Depending on your database type, you can do a database dump or database transaction log dump.

If you are using MySQL database, the command is mysqldump. But mysql don't give you incremental backup on the community edition. However there is the binlog approach that is used for replication to slave db.

For postgresql, you can use the WAL. They are originally used for replication too.

However you have to take note if your solution allows for update on both sides. That will create synchronisation problem.

 

[AshuraSg]

If there is no network connectivity from source to target, then you can forget about any realtime or sub realtime replication or synchronisation techniques.

Depending on your database type, you can do a database dump or database transaction log dump.

If you are using MySQL database, the command is mysqldump. But mysql don't give you incremental backup on the community edition. However there is the binlog approach that is used for replication to slave db.

For postgresql, you can use the WAL. They are originally used for replication too.

However you have to take note if your solution allows for update on both sides. That will create synchronisation problem.

Hi David,

Thanks for the reply. I think I complicated my question haha sorry about it. Basically what we are looking at is to ensure fast data transfer from one location to another. Other than having direct fiber optic connection, we would like to see if there is any other technology that can achieve such scenario and at a faster speed. Sorry again for misleading.

 

[davidktw]

Hi David,

Thanks for the reply. I think I complicated my question haha sorry about it. Basically what we are looking at is to ensure fast data transfer from one location to another. Other than having direct fiber optic connection, we would like to see if there is any other technology that can achieve such scenario and at a faster speed. Sorry again for misleading.

There is no need for high bandwidth fibre optics unless you are talking about 4Gbps or more consistent bandwidth required.

Even if you have data input in your old database server in adhoc manner, I highly suspect it will be unlikely you hit 1Gbps on the faithful 1Gbps copper ethernet.

If you can just wire up or old room and new room using LAN using a simple L2 switch, then you can easily configure your database to replicate from the old server into the new one.

The only constraint is you MUST NOT ALLOW any update on the slave db server (in the main data centre). That is how replication works.

The initial data dump from the old db(primary) server to the new db(slave) server can be done out of band by dumping out the database data and import into the new database. After which you configure the master-slave relationship between the 2 database servers, after which the replication will take care of the data consistency.

This setup is not simple, but it is practice by the industry extensively.