Blog from September, 2008

The Project IT Infrastructure Development (PITID) is a University Project to improve Jacobs' IT infrastructure by moving to centralized storage, employing virtualization technologies, and focusing on a reduced set of infrastructure elements.

For Jacobs' users this will lead to a single login for most services, faster and more reliable communication means including all Jacobs members, and extended offers like home directories for everyone.

(thumbs up) After long preparation we are now ready to move on to contract signing and ordering.

Upcoming Changes

According to current planning, most of the noticeable changes will be implemented in January 2009.

Here is an updated (vs. the last announcement) overview:

On Thursday 25 September 2008 between 07:00 and 08:00 local Bremen time servers and network components will be updated.

Server and network operations will be unreliable. Expect unavailable servers and sudden downtimes of services without prior explicit announcement.

If you have to work on files stored on the server during that time, then before the maintenance hour copy the respective files to your local hard disk, work on it locally, and after finishing your work and after the end of the maintenance interval, copy them back onto the server.

For more details on the maintenance hours see the Maintenance Hours page.

The Server Admin02 encountered a hardware failure wich could not be repaired.

All services (e.g. printing, file access (public folder), etc.) are now handled by Admin01 (warning)

The public folder is from now on at \\admin01\public\ (warning)

All printers are accessible via Admin01 ( \\admin01\ ) (warning)

No data loss occured (tick)

Please update your shortcuts, drive mappings, and printers. See How to add network printer to Windows client computer for help on network printer selection.

If access to the directory \\admin01\public via a network drive existed before the issue, please disconnect the network drive, and connect it again to get rid of old drive mapping information.

On Thursday 11 September 2008 between 07:00 and 08:00 local Bremen time servers and network components will be updated.

Server and network operations will be unreliable. Expect unavailable servers and sudden downtimes of services without prior explicit announcement.

If you have to work on files stored on the server during that time, then before the maintenance hour copy the respective files to your local hard disk, work on it locally, and after finishing your work and after the end of the maintenance interval, copy them back onto the server.

For more details on the maintenance hours see the Maintenance Hours page.

Go Bananas not reachable

Currently, the free dial-in numbers of Go Bananas are not reachable from Jacobs University telephones.

We are investigating.

Update

0800 100 61 72 apparently works.

The IT Student Support has now been established and is available during:

Days

Morning

Afternoon

Monday to Friday

10:00-12:00

13:30-15:30

For details, see Service Desk for Students

Jacobs Blackboard created

The beginning of the semester lead to new heights in communication within the Jacobs community. The overwhelming mass of broadcasted emails, mainly caused by uncritical use of the mailing list "jacobs-students", resulted in delivery delays of up to several days. As this doesn't allow for a reasonable dialogue anymore, the mailing list "jacobs-students" will be deactivated for the time being, held emails will be delivered as soon as possible.

Instead, to allow for more efficient and modern communication within the Jacobs community and beyond, an electronic "Jacobs Blackboard" has been created at http://teamwork.jacobs-university.de/display/jacobsbb/Home

Everyone at Jacobs can start a discussion, and discussions and comments can be read in a browser or followed by subscribing to RSS feeds. Events and any other "small talk" at Jacobs can be posted there as well.

This development and possible improvements will be discussed in the next joint meeting of the Academic Council and the Student Government.

Excited about the development,
Torge Schmidt, CTO

Server maintenance 2nd&4th Thursday each month, 07:00-08:00.

IT infos and blog at http://teamwork.jacobs-university.de/display/ircit/.
Regarding IT support see http://teamwork.jacobs-university.de/x/iQ.
For CampusNet support send campus.net@jacobs-university.de.

Mail server going crazy

Due to heavy spamming posting on various email lists, esp. jacobs-students and jacobs-events, the mail server goes wild at the moment:

Current average load is 164 mails per minute worked on. As each mass email is broken into chunks of 30 recipients per email, that is a rate of 4920 emails worked on per minute, or 82 per second!

Adding to this 374 emails per minute or (times 30 divided by 60 seconds per minute) 187 per second for non-deliverable emails (usually, because the mailbox is over quota!) adds to a whopping 269 emails handled PER SECOND!

That has impact on the overall email performance, which is why IRC-IT is currently throttling down mass emails to give the server some slack for delivery of other emails.

The current email system is scheduled to be replaced by a new one, which will allow even faster spamming information distribution. Unfortunately, it is not available yet.

An alternative to sending emails would be using

for information distribution. Then, a piece of information is not replicated to 1200+ mailboxes, but instead only uploaded once!

Update 2008-09-02 09:00

After mass emails were throttled at around 3pm yesterday, immediate message could all be delivered, and mass emails were slowly rescheduled, as can be seen in this graph:

Also note, that the mean was 1436 emails per minute worked on, that is (times 30 divided by 60) 718 emails per second!

Update 2008-09-02 12:00

The mail queue goes through the roof:

We are throttling jacobs-students again. Also, Email retrieval is reduced to only once per 4 minutes, which might lead to error messages upon earlier retrieval. Ten minutes is a recommended value.

Update 2008-09-02 16:30

After holding almost 100.000 emails on jacobs-students the email system has gone back to normal:

Mails will be released slowly during the next hours.

Actually, it's not the mail server which goes crazy! If this crazyness persists, we will have to take down the mailing list. Please be reasonable in using Jacobs' resources!

Update 2008-09-03 10:00

After releasing emails yesterday at 16:30, the queue built up again until emails to jacobs-students were held again at around 19:00, when operation turned back to normal. All helt emails have been released at around 22:00, then the delivery queue built up to almost 7000 entries, times 30 emails per entry, that's about 210,000 emails to deliver over night, plus new incoming. As this didn't compete until today in the morning, emails to jacobs-students were held again at around 08:00 today. The queue slowly builds up again due to new incoming emails as well as posts to jacobs-events:

Update 2008-09-03 17:30

The mail queue has been kept at a somewhat reasonable rate during the day by manually postponing emails of the jacobs-students email list:

6,598 queue entries (amounting to almost 200,000 emails on jacobs-students alone!) will be released in the later evening for delivery through the night. It should not be expected, that all emails can be delivered until tomorrow morning, when delivery will be stopped again.

Update 2008-09-05 13:45

Currently 11844 queue entries (amounting to ~355.000+) emails are held for delivery. Even releasing everything during the night is not sufficient for getting the number of held emails down:

Update 2008-09-08 11:45

After no longer accepting messages on the jacobs-students, the number of messages waiting for delivery were reduced to ~30.000 over the weekend:

Unfortunately, another 170+ emails are waiting to be distributed, amounting to 200.000+ single emails to be delivered! In addition, email traffic shifts to other student lists, with cross-posting. Hence, we are currently also holding other student email lists with more than 3.000 emails an hour ago. (sad)

(lightbulb) Please use jacobsbb as an alternative to mass-mailing.

Update 2008-09-10 14:00

Yesterday saw interesting developments. The mail queue was empty at around 04:30 in the morning. At around 09:30 several mass emails hit the server and mail delay rose until 12 o'clock. At that time, more than 50,000 emails had been delivered on that day.

During the lunch break more emails could be delivered than newly arrived, but starting at 13:00, the q length rose again. Until 15:00, when the Land der Ideen award ceremony happened, but starting shortly after that, the mass mails continued and the queue length rose again.

Today in the morning the queue was empty, all non-moderated emails were delivered. During the day, several mailing list emails were sent through the server, but were within normal capacity. At the moment, more than 45,000 emails have successfully been delivered into users' mailboxes.

Mail run time and delivery delay is in the range of seconds at the moment.

(lightbulb) Using the jacobsbb has saved many thousand (mostly unread?) emails! Thanks for adopting, and please continue using it!

Update 2008-09-16 10:30

The following graph shows the commotion of the last two weeks, and how it subsided:

A policy is being drafted to define appropriate use of mailing lists in the future. All emails held during emergency moderation had been rejected to allow for a clean start. The mailing list jacobs-students is currently run by the Student IT Committee. The jacobsbb is being used for various messages which previously were sent by email, saving a lot of traffic and storage, and is open to the whole Jacobs community.

This is an automatically generated measurement of key performance indicators (KPIs) of Service Level Agreements for August 2008.

The list of other KPI measurement reports is at KPI Measurements.

Summary Table

KPIs Specified

Not Measured (warning)

Targets Met (plus)

Targets Failed (minus)

16

1

13

2

KPI Measurements

If the specification is met, a (plus) is in the Met column, else a (minus); KPIs not measured have a (warning) there.

Host

Service

SLA

Target

Measured

Met

facultyweb

FTP

Faculty Web Service

99.000000%

100.0%

(plus)

 

HTTP

Faculty Web Service

99.000000%

100.0%

(plus)

 

HTTPS

Faculty Web Service

99.000000%

100.0%

(plus)

login

SSH

Remote Login Shell Service

99.000000%

100.0%

(plus)

alea

HTTP

ALEA Server Service

99.000000%

100.0%

(plus)

campusweb

HTTPSLOGIN

CampusNet Server Service

99.000000%

99.165%

(plus)

admissionweb

HTTPSLOGIN

CampusNet Server Service

99.000000%

(not measured)

(warning)

apps01

HTTP

CampusNet Server Service

99.000000%

100.0%

(plus)

apps02

HTTP

CampusNet Server Service

99.000000%

100.0%

(plus)

apps03

HTTP

CampusNet Server Service

99.000000%

100.0%

(plus)

kastor

DISK_GLOBALDB

CampusNet Server Service

99.000000%

20.972%

(minus)

pollux

DISK_GLOBALDB

CampusNet Server Service

99.000000%

95.655%

(minus)

helena

HTTP

Alumni Email Server Service

99.000000%

100.0%

(plus)

 

POP3

Alumni Email Server Service

99.000000%

100.0%

(plus)

 

IMAP

Alumni Email Server Service

99.000000%

100.0%

(plus)

 

SMTP

Alumni Email Server Service

99.000000%

100.0%

(plus)