This distributed data acquisition system use optical fiber to implement precise clock distribution and data/instruction transmission at the same time. It is designed for low frequency radio astronomy telescope (China’s 21CMA) to replace their analogy beamforming to digital beamforming and improve their system performance.
I built a verification system simulating two nodes, one local node and one remote node. The system contains two FPGA board (Xilinx KC705), two ADC board (ADI AD9361), two clock jitter cleaner board (TI LMK04906) and some other support hardware. I design and implement the entire FPGA software and write part of the driver and windows application on the PC side.
The highlight of this system is that i design a digital phase lock method to compensate the length drift of the optical fiber because of the change of temperature and hugely decrease the long-term clock jitter in the remote side. We test the clock synchronization performance between the local and remote node using this method, the short-term jitter is 1.2ps RMS and the long-term jitter is 5ps RMS. The optical fiber is 200 meter in length and the temperature changes dramatically (at least 30 degree in range). I have been writing a patent based on this method and it will be published soon.
Below is the basic diagram of the precise clock distribution and data/instruction transmission using optical fiber.
Figure 1-1 Precise clock distribution using optical fiber diagram
Figure 1-2 Data and instruction transmission using optical fiber diagram
Figure 3 is the block diagram of the verification system and the testing method. The basic idea is as follow. We used the signal generator and splitter two generate two identical input signal. This two identical signal were input to the local and remote ADC sampling board. We control the system to sampling the signal at the same time (through the local node) and store the sampling data in the DDR3 memory in the FPGA mother board. Then we transmitted the remote node sampling data to the local node. Finally, we transmitted the local/remote sampling data to the control/analysis PC through PCI-Express interface and analyse the short-term and long-term clock jitter using FFT. Figure 4 is the real world system and testing environment.
Figure 1-3 Block diagram of the verification system and testing method
Figure 1-4 Real world system and testing environment
2, Drift compensation method
The basic idea of the digital phase lock method to compensate the length drift of the optical fiber because of the change of temperature is as follow. We used the reciprocity of the round trip of the optical fiber transmission, which means the transmit link and the receive link of the optical fiber changed simultaneous. Thus, we used a phase detector to detect the phase difference between the local node optical fiber receive clock and the local node source reference clock (local sampling clock in practical). Then we digitalized the phase difference using the XADC in FPGA. We then calculated the compensate phase difference to the remote node through the change of the phase difference (caused by the length drift of the optical fiber) in the FPGA. At last we change the local node optical fiber transmit clock using a Direct Digital Synthesizer (DDS) to compensate the clock jitter in the remote side.
Here is the basic mathematical expression. Let be the local node sampling clock or the source reference clock. be the local node sampling clock. be the local node optical fiber transmit clock and be the local node optical fiber receive clock. be the phase difference between the local node and the remote node of the optical fiber. It will change with the length of the optical fiber and cause clock jitter of the remote node. Assuming the transmit link and the receive link are the same (transmit and receive link optical fiber are almost the same length). We have:
Assume then we got :
The above expression constitutes a close loop. stands for the phase difference detected by the phase detector. stands for the phase difference between the local node optical fiber transmit clock and the local node sampling clock (controlled by the DDS).
The block diagram of the digital phase lock drift compensator is as follow.
Figure 2-1 Block diagram of the digital phase lock drift compensator
3, FPGA system level design and implementation
I designed and implemented the entire FPGA software. I also take care of the entire debugging (both hardware and software) and testing process.
The FPGA software system contains five main modules. 1, real-time data sampling module. 2, optical fiber transmission control module (using CPRI IP). 3, memory control module. 4, PCI-Express interface module. 5, digital phase lock processing module. The remote node and the local node FPGA system are almost identical except remote node didn't contain module 4 and 5. Below is the block diagram of top level of the FPGA software system.
The FPGA software system contains mainly 4 clock domains. 1, real-time sampling data clock domain (200MHz). 2, optical fiber transmission clock domain (153.6MHz) 3, DDR3 interface clock domain (200MHz). 4, PCI-Express interface clock domain (250MHz).
Figure 3-1 FPGA system top level block diagram
Below is the device utilization summary of the FPGA (xc7k325t-2ffg900) in the local node. The remote node FPGA utilization is fewer.
3.1 Real-time data sampling module
The AD9361 PHY module was transplanted from the ADI design demo in github, which including a Microblaze soft core executed the ad9361 physical driver (in C code).
Figure 3-2 Block diagram of the real-time data sampling module
3.2 Optical fiber transmission control module
The Optical fiber transmission control module used CPRI IP to achieve optical fiber physical interface and it has three basic functions. 1, clock control and distribution. 2, data transmission. 3, system control and monitor instruction transmission. This was implemented by a self-design data/instruction transmission protocol.
Figure 3-3 Block diagram of the optical fiber transmission control module
3.3 Memory control module
The memory control module used a static priority control data scheduling method to control the six basic data streams in real - time. The memory control module also contains a DDR3 interface module using Xilinx MIG IP.
Figure 3-4 Block diagram of the memory control module
3.4 PCI-Express interface module
The PCI-Express interface module was programmed in TLP level. Using Memory_Rd Memory_WR Register_RD Register_WR TLPs.
The PCI-Express interface module and the Memory control module made up a basic Direct-Memory-Access (DMA) system with the host PC. The DMA minimum read and write size is 4096 Byte which is the same length as the real-time data frame size.
Figure 3-5 Block diagram of the PCI-Express interface module
3.5 Digital phase lock processing module
Below is the detailed implementation diagram of the digital phase lock processing module in the local node which is the main part of the drift compensation method.
Figure 3-6 Block diagram of the digital phase lock processing module
4, Testing results
Below is our host application gui in the PC side. It was implemented using windriver (PCIE driver) and VC 6.0 (MFC). The host application realized the basic control of the system, including system work on, reset, data sampling (local and remote node), data storage, data transmission (through optical fiber and PCI-Express). And some basic testing function about optical fiber data transmission (bit error rate).
We design a automation testing method using the automation function in the MFC with matlab to implement testing. The testing script was written in matlab code, controlling the host application to control the system and transmit the data. At last, the matlab analysed the data and draw the figures.
Figure 4-1 Host application GUI
4.1 Distributed clock quality in the remote node
This part mainly tested the short-term performance of the clock distributed in the system. We measure the phase noise of the remote node sampling clock using a spectrum analyzer. The phase noise is below 92dBc at 10kHz.
Figure 4-2 remote node distributed clock phase noise:>92dBc (10KHz) spectrum analyzer parameters : SPAN:25KHz,BW:1Hz
4.2 Data transmission BER
We test the transmission BER several times in the condition of a 200 meter length optical fiber and didn't find error bits.
|test No.||test time (30min)||transmission rate(Gbps)||BER|
4.3 Distributed clock synchronization performance test
The clock synchronization performance or the clock jitter performance test is the core performance of this system. It directly determines the sampling performance of the distributed acquisition system. The highlight feature of our system, i.e. drift compensation method hugely improve the clock synchronization performance. Here is the comparison test between not using this method (open loop) and using this method (close loop).
The test environments are as follow. The optical fiber was 200 meter in length. The optical fiber was heated uniformly by a drier and cool down naturally to simulate the temperature drift in the real world. The input signal to the ADC are 2401Mhz and the LO of the ADC is 2400MHz (reference by the distributed clock recovery from optical fiber). We sample the data in the local node and the remote node at the same time every 15 seconds and analysed the data. We calculate the phase difference in average of the data from the two node and finally converted into time unit (picosecond) i.e. clock jitter. In the figure below y axis is the phase difference in 2401MHz.
Figure 4-3 clock synchronization performance (open loop)
long-term clock jitter peak to peak : 289.8391ps
long-term clock jitter RMS : 60.3373ps
Figure 4-4 clock synchronization performance (close loop)
long-term clock jitter peak to peak : 22.6991ps
long-term clock jitter RMS : 3.9427ps