From Digitalis

(Difference between revisions)
Jump to: navigation, search
(Add system changelogs)
Line 202: Line 202:
* Getting the highest bandwidth possible out of your Phi application:
* Getting the highest bandwidth possible out of your Phi application:

Revision as of 16:07, 10 December 2013

This page describes the first steps to play with the Xeon Phi installed in the Digitalis platform in idphix.



A Xeon Phi 5110P is installed in the idphix machine

The setup is the following:

We use an internal network on the host (idphix) for the Xeon Phi:

This gives the following scheme:

                        | -------------             -----------  |
                        ||    host     |           |  Xeon Phi | |
                        ||   idphix    |           |idphix-mic0| |
Grid5000 network <---> eth0           mic0 <----> mic0         | |
               | |
                        ||             |           |           | |
                        | -------------             -----------  |

How to experiment ?

For now (2013-09-13) the operating system is a Centos 6 (OS officially supported by Intel for the Xeon Phi).

Use the machine as follows:

Connect to digitalis/grid5000

workstation$ ssh digitalis.grenoble.g5k

In case of any issue (network issue, connection hanging...) please read more info here or ask on the Digitalis mailing list.

Reserve the machine/run a job

digitalis$ oarsub -p "machine='idphix'" -l walltime=2 -I

Setup your access to the Xeon Phi

(This might not be relevant if your are using some level of abstraction (COI, ...)

Make sure you created your G5K ssh keys (keys without passephrase protection, dedicated to G5K and stored in your NFS home directory)


idphix$ sudo /usr/local/bin/mic-setup-my-user

If you get the error:

Error: Cannot get your SSH keys, please run 'chmod 755 ~/.ssh && chmod 644 ~/.ssh/{authorized_keys,id_*.pub}'

Please run:

idphix$ chmod 755 ~/.ssh && chmod 644 ~/.ssh/{authorized_keys,id_*.pub}

Output should be:

idphix$ sudo /usr/local/bin/mic-setup-my-user 
*** Configure user pneyron:users (10106:8000) on mic0
*** Copy user files to mic0                                    100%  420     0.4KB/s   00:00    
authorized_keys                               100%  916     0.9KB/s   00:00    
.profile                                      100%   62     0.1KB/s   00:00    
*** Force creation of pneyron:users on mic0 and fix file ownership and permission

You shoud now be able to ssh to the coprocessor:

idphix$ ssh mic0
idphix-mic0$ uname -a
Linux #2 SMP Tue Apr 30 14:05:06 PDT 2013 k1om k1om k1om GNU/Linux

Intel compiler, compilation for the MIC

Intel compiler (icc & co) is installed in /opt/intel /applis/digitalis/intel.

It is configured to use Inria's license server (FlexLM).

This licence server provides 2 tokens which are shared among all Inria's users. Only 2 compilations can be performed at a time.

If icc complains about the license (error message talking about FlexLM), please report the issue to

Rebooting the MIC

If needed, you can reboot the MIC from within a exclusive job (doing so in shared job session would risk annoying other users).

oarsub -t exclusive -p "machine='idphix'" ....

Then, you will be able to run

sudo /usr/local/bin/mic-reboot

(no password should be requested).

Rebooting the MIC takes some time. You can look at the output of dmesg or miccrtl -s meanwhile.

Data on the mic: no persistence

Please mind the fact that data pushed to the mic are not persistent. Your home directory on idphix-mic0 could be emptied for various reasons.

System changelogs

idphix-default 20130926

  • Centos 6
  • Linux 2.6.32-358.el6.x86_64
  • MPSS packages
  • Intel Cluster Studio XE 2013
  • MIC info
[root@idphix ~]# /opt/intel/mic/bin/micinfo 
MicInfo Utility Log

Created Mon Dec  9 20:13:37 2013

	System Info
		HOST OS			: Linux
		OS Version		: 2.6.32-358.el6.x86_64
		Driver Version		: 6720-13
		MPSS Version		: 2.1.6720-13
		Host Physical Memory	: 66077 MB

Device No: 0, Device Name: mic0

		Flash Version 		 :
		SMC Firmware Version	 : 1.14.4616
		SMC Boot Loader Version	 : 1.8.4326
		uOS Version 		 :
		Device Serial Number 	 : ADKC32100375

		Vendor ID 		 : 0x8086
		Device ID 		 : 0x2250
		Subsystem ID 		 : 0x2500
		Coprocessor Stepping ID	 : 3
		PCIe Width 		 : x16
		PCIe Speed 		 : 5 GT/s
		PCIe Max payload size	 : 256 bytes
		PCIe Max read req size	 : 512 bytes
		Coprocessor Model	 : 0x01
		Coprocessor Model Ext	 : 0x00
		Coprocessor Type	 : 0x00
		Coprocessor Family	 : 0x0b
		Coprocessor Family Ext	 : 0x00
		Coprocessor Stepping 	 : B1
		Board SKU 		 : INVALID SKU
		ECC Mode 		 : Enabled
		SMC HW Revision 	 : Product 225W Passive CS

		Total No of Active Cores : 60
		Voltage 		 : 985000 uV
		Frequency		 : 1052631 kHz

		Fan Speed Control 	 : N/A
		Fan RPM 		 : N/A
		Fan PWM 		 : N/A
		Die Temp		 : 35 C

		GDDR Vendor		 : Elpida
		GDDR Version		 : 0x1
		GDDR Density		 : 2048 Mb
		GDDR Size		 : 7936 MB
		GDDR Technology		 : GDDR5 
		GDDR Speed		 : 5.000000 GT/s 
		GDDR Frequency		 : 2500000 kHz
		GDDR Voltage		 : 1501000 uV

idphix-default 20131209

  • Updated Centos packages, except linux kernel
  • Moved Intel Cluster Studio XE 2013 to NFS /applis/digitalis/intel
  • No change to the MIC software (MPSS)


Personal tools