Site problem 12/04/2023

...

I also have records lost on 12-Apr-23.  After the iRecord site came back in the afternoon from maintenance, two things went wrong with my records. 

1) A verifier had re-determined the species of a mollusc entered on 11-Apr-23, leading to a resulting notification. I looked at the notifications and first navigated to the record using the link in the notifications. The record was clearly there to view and so was the verifier comment. The verifier had made his re-determination and comment, i.e., an edit to the record, on the afternoon of Wed. 12-Apr-23, an hour or so after iRecord came back from maintenance.  I accepted the notifications and the site acknowledged the acceptance. Since then, around mid-afternoon on Wed. 12-Apr-23, I have not seen the mollusc record. 

2) Waited a few more hours before doing anything else on iRecord as I had other calls on my time. During the evening of 12-Apr-23, after 21.00 or so, I entered a new record of 4 mallard feeding. iRecord confirmed the submission had been successful. I had expected the new record to appear in both Explore My Records and Explore All Records after 15 minutes, os so, as usual, but it did not appear over the next few hours. The new record of mallard feeding has still not appeared the following morning. 

The impression given by this behaviour by the site is that both existing records updated and new records entered since iRecord came back from maintenance on the afternoon of Wed. 12-Apr-23 have gone missing. They are simply not available to view. 

The impression that new records entered are lost is compounded by the results returned by the Explore All Records report. When one simply navigates to the page and lets it run automatically, it returns the same result at 10.20 on Thurs. 13-Apr-23 as it did when it came back from maintenance on the afternoon on Wed. 12-Apr-23. The first record is a record of a badger (meles meles) seen in the Vice County Fifeshire and entered by Derek Crawley. iRecord no 29893046. The record sticks in my mind as Derek Crawley is the founder of the Mammal Society, so his name is easy to remember on iRecord. He entered the record on the morning of Wed. 12-Apr-23 at 08.24, which might indicate the site was still available for data entry at that point. What iRecord appears to be doing is showing the most recent record entered as a record entered by Derek Crawly at 08.24 on Wed. 12-Apr-23 when it is now 26 hours later, at the time of writing, and so there should be lots of other records entered in those 26 hours that should appear before Derek Crawley's. For example, the record showing four mallard feeding that I entered on the evening of 12-Apr-23. I expect there will be many other records entered by many other people that are also missing. 

I have written such a detailed description of the problem and symptoms in order to make it easier for programmers to identify the problem and fix it. I would encourage other users to enter their stories of missing records too, as that can help the programmers find out what has gone wrong. 

Would be grateful if the programmers or iRecord support would reply to this thread once they have fixed the problem.

I have tried several times to even add to this thread, but nothing works....

I didn't expect that to work, but it did. So I have at last successfuly managed to add to the comments on this thread, which is progress.

However I can now add that I too have lost records added yesterday evening, and this morning. I have also altered 2 records from genus to species level. The simple list of all my species still shows the record unchanged, but the details page does show the update is now done.

I do not trust this system. I just hope it gets fixed soon.

Thanks for raising these issues. There is a currently a problem with records being displayed in iRecord, but they are being saved in the underlying database. The latest records can be seen by going to the old version of the Explore all records page or the old version of the Explore my records page.

When records are added to iRecord they are immediately stored in the database. However, to be displayed on the main Explore and Verification pages they need to be transferred into a separate view of the records, which uses ElasticSearch (ES) technology. Normally the transfer to ES takes about 10-15 minutes. However, yesterday's large update to our species dictionary has caused a long queue in the transfer to ES and the system is still catching up, hence the delay in records appearing on the ES pages.

Our developers are reviewing this and we will update you as soon as we can on when the system will catch up and display the records as usual.

Apologies for the inconvenience. If you have entered records that have not yet appeared on the normal Explore pages please check the old versions of the Explore pages via the links above, to avoid re-entering records that have already been saved.

Our developers have made some tweaks and it looks as if the recent records are moving back into the ES view more rapidly, and the normal Explore pages are now showing today's records. There are still further species dictionary updates going on in the background so it may be a few more days before everything is fully back to normal. If anyone is still experiencing loss of records or other oddities please send us the details via irecord@ceh.ac.uk, including any record ID numbers if they are available.