forked from DIRACGrid/DIRAC
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathrelease.notes
10988 lines (7952 loc) · 381 KB
/
release.notes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
[v7r3p38]
*WorkloadManagement
FIX: (#7122) reintroduce 'Staging' to 'Checking' transition in JobsStateMachine
NEW: (#7118) Introducing 'Scouting' status in job state transitions
[v7r3p37]
*WorkloadManagement
FIX: (#7043) fix(StalledJobAgent): after a job was Killed, we have to force the Failed.
FIX: (#6972) JobStatus: add staging to checking transition, needed by StorageManagementSystem (Stager)
NEW: (#6963) tests for RemoteRunner
CHANGE: (#6963) RemoteRunner can deal with MP applications, clean job outputs and process more applications
FIX: (#6963) Modify the DIRACVersion in the PushJobAgent
FIX: (#6987) return specific exit code in RemoteRunner in case of failure
FIX: (#6984) Getting the value of /LocalSite/RemoteExecution
*Resources
FIX: (#7037) add workaround for https timeout when dowloading file
FIX: (#6990) ARCLogLevel
NEW: (#6963) cleanJob() in AREX
CHANGE: (#6963) executables from inputs are passed as executables in the XRSL in ARC and AREX
*docs
CHANGE: (#6963) more details in the "supercomputers" section
[v7r3p36]
*Core
FIX: (#6807) don't use matplotlib deprecated methods
FIX: (#6762) Fix exception in FileHelper.networkToString on Python3
*ConfigurationSystem
FIX: (#6866) VOMS2CSAgent: strip whitespaces from DN entried and nickname
*Interfaces
NEW: (#6850) Add setPriority to Job API
*RequestManagementSystem
CHANGE: (#6844) ReqProxy does not attempt a direct forward to central ReqManager
CHANGE: (#6741) adapt ReqDB to sqlalchemy 2.0
*Resources
FIX: (#6840) Typo in AREX error meesage
FIX: (#6837) Complete AREX renewal
FIX: (#6828) Update AREX delegation renewal
FIX: (#6826) Fix AREX ARCRESTTimeout
FIX: (#6813) fix account parameter in SSHBatchCE
FIX: (#6766) better handle exceptions in AREXCE
*WorkloadManagement
FIX: (#6766) config helper error message
FIX: (#6747) Reorder pilot downloads to minimise race condition
*DataManagementSystem
FIX: (#6816) specify the same column in order_by and distinct in FTS3DB.getNonFinishedOperations
CHANGE: (#6741) adapt FTS3DB to sqlalchemy 2.0
*Tests
NEW: (#6745) Integration test for StorageElementHandler and FileHelper
[v7r3p35]
*Resources
FIX: (#6728) Add a note about XRSL times in ARC(6) vs AREX
FIX: (#6676) Still report AREX statuses if renew/cancel fails
FIX: (#6661) Use full jobID in AREX jobsToCancel
CHANGE: (#6656) try yet another value for Echo removal timeout (their request)
CHANGE: (#6646) Documentation of the CEs
FIX: (#6624) SSHCE getJobOutput()
FIX: (#6624) SLURM getJobOutput() when multi-node allocation and that the job is killed
FIX: (#6615) convert ARC to DIRAC JobID in AREXCE
CHANGE: (#6604) drop the WholeNode CE option from Condor
*Core
FIX: (#6720) Fix in FileHelper module writing to file, fixed exception "TypeError: write() argument must be str, not bytes, because file was opened in the wrong mode.
NEW: (#6643) Support OpenSSL 3.0.x
*FrameworkSystem
FIX: (#6714) ComponentSupervisionAgent: Fix problem if two instances in different systems but with the same name are present on the same machine (e.g., Framework/Monitoring and Monitoring/Monitoring)
*WorkloadManagementSystem
FIX: (#6713) Matcher handles cases of empty Tag in resource description
FIX: (#6602) ExtraPilotOptions can be a comma-separated list
FIX: (#6660) Handle exceptions for ThreadPoolExecutors
*Accounting
FIX: (#6693) JobsPerPilot plot when no data available
*Interfaces
FIX: (#6665) More permissive filter in job kill/delete client API
FIX: (#6632) dirac-admin-add-host: fixed to allow use of long option names
*TransformationSystem
FIX: (#6623) Race condition in TransformationDB.__addDataFiles
*StorageManagementSystem
FIX: (#6610) fix py3 incompatibility in wakeupOldRequests
[v7r3p34]
*Resources
FIX: (#6584) fix delegation process in the AREXCE
FIX: (#6539) multi-node allocations with SLURM
FIX: (#6535) AREXCE._getDelegation() returns an error if the process cannot be completed
[v7r3p33]
*Resources
NEW: (#6526) add the ARCLogLevel option in ARC
NEW: (#6526) add the ComputingInfoEndpoint in ARC6
FIX: (#6526) restore a commented line in AREX._reset()
FIX: (#6516) Support using ARC that was compiled against SWIG 4.1+
NEW: (#6502) Added network selection to CloudCE.
*DataManagementSystem
FIX: (#6518) FileCatalogCLI: ls command: Added a missing else to avoid failing
silently if directory is not found as opposed to it just being an empty directory.
*WorkloadManagementSystem
FIX: (#6514) Ensure cpuNormalizationFactor is a float
*docs
FIX: (#6507) Added CloudComputingElement to AdminGuide/Resources
[v7r3p32]
*WorkloadManagementSystem
NEW: (#6477) backported ElasticJobParametersDB from rel-v8r0 branch
FIX: (#6469) reduce the PollingTime of JobAgent to 20s
*Core
FIX: (#6468) quoting the DB passwords for special characters
[v7r3p31]
NEW: (#6423) Information about VirtualOrganization parameter added.
*WorkloadManagementSystem
FIX: (#6422) WMSUtilities.killPilotsInQueues have a consistent return value
*Core
FIX: (#6395) Use safer mode for grid-security directories
[v7r3p30]
*Resources
CHANGE: (#6372) negative free space value is transformed to 0
*DataManagementSystem
NEW: (#6371) cancel the FTS transfers when an RMS request is found to be canceled
[v7r3p29]
*Core
CHANGE: (#6325) Set OpenSSL ciphers to "DEFAULT:@SECLEVEL=1"
NEW: (#6325) Experimental support for Python 3.10
NEW: (#6325) Suppress InsecureRequestWarning when using dirac-configure with HTTPS servers
*Resources
FIX: (#6358) new directory creation specifies the permission in octal mode
NEW: (#6352) Add a setGfalSetting decorator
CHANGE: (#6352) add a 30s timeout for removal at RAL
*WorkloadManagementSystem
FIX: (#6353) Stop instance from VirtualMachineMonitor
NEW: (#6348) The default InputDataDirectory used by DownloadInputData can be overridden by
setting /LocalSite/InputDataDirectory
[v7r3p28]
*Core
CHANGE: (#6324) Replace suds-jurko with suds
*Resources
FIX: (#6319) Forward jobDesc to innerCE in PoolCE
FIX: (#6310) Set HOME in SingularityCE
FIX: (#6303) Return proper error if provider value missing in CloudCE
*FrameworkSystem
FIX: (#6305) Don't notify users with expired proxies
*WorkloadManagement
FIX: (#6304) Log error if CE Platform is missing.
[v7r3p27]
*ConfigurationSystem
CHANGE: (#6284) Make GOCDB2CSAgent functionMap less private
[v7r3p26]
*Resources
FIX: (#6274) Conditional FC ProxyPlugin checks the ThreadConfig
*DataManagementSystem
FIX: (#6266) Fix FC change{group,owner,mode} for multiple LFNs
[v7r3p25]
*DataManagementSystem
FIX: (#6266) Fix FC change{group,owner,mode} for multiple LFNs
FIX: (#6238) DatasetManager: bug fix in getDatasetsInDirectory() method
*WorkloadManagement
NEW: (#6252) ContainerBind option can be set on the per-site and per-CE level
FIX: (#6236) Fix Platform key error in SD when CheckPlatform=True
*FrameworkSystem
NEW: (#6237) Add StdoutJsonBackend logger backend
*Resources
FIX: (#6235) remove REST information from the ARC6 pilot job references to bind them with jobs
FIX: (#6221) dirac_resource_info - restored the SE lookup logic
*Core
FIX: (#6225) Increase VOMS command timeout
*Interfaces
CHANGE: (#6224) Make tmpdir for local cache instead of using CWD directly when replicating
[v7r3p24]
*Resources
FIX: (#6217) ARCCE.getARCJob() becomes protected
FIX: (#6213) ARC6 typo issues
FIX: (#6213) ARC writeXRSL becomes protected
NEW: (#6199) Adapt MQConnector to stomp 8
*DataManagementSystem
FIX: (#6207) Built-in metakeys were incorrectly modified in a MultiVO environment. This fix keeps them intact while modifying user-supplied metadata names by append a VO suffix, as before.
*ResourceStatusSystem
FIX: (#6191) convert ComputingElement to CE before searching in the configuration
[v7r3p23]
*CI
NEW: (#6187) max open files for dirac CI container to 8192
*WorkloadManagementSystem
CHANGE: (#6183) new method to submit pilots from ARC
FIX: (#6133) use the JobMonitoringClient to check for job parameters (that can be in the ES backend)
FIX: (#6101) Avoid producing massive log output in OptimizationMind
*Core
FIX: (#6155) Replaced old vomses/vomsdir code to be able to handle cvmfs vomses. See #6150
FIX: (#6123) fix error reporting in TornadoService
FIX: (#6118) don't truncate first character in cli doc strings
*RequestManagementSystem
NEW: (#6148) Add option to automatically cancel Scheduled requests after a defined time period.
Default behaviour unchanged. Relates to #6136
*Resources/Cloud
FIX: (#6145) userData need to be in "ascii" string format to meet the requirement of "create_node"
when creating VMs from cloud resources
*DataManagementSystem
NEW: (#6113) Introduces an option 'ForceIndexedMetadata' to disable the creation of unindexed metadata.
Default: False, i.e. preserves the current default where both indexed and unindexed metadata are allowed.
Relates to #6029.
*Resources
FIX: (#6105) Error in the condition to get FlavorName
FIX: (#6100) use identity instead of subject of proxy in CloudCE
*docs
NEW: (#6090) add info on installing non-released code from within the pilot
[v7r3p22]
*Resources
FIX: (#6082) autodetect cvmfs proxy in default cloudce template
FIX: (#6079) use ceType instead of ceName for CE parameters
FIX: (#6073) ARCCE make ldapsearch call forward compatible with newer ldapsearch versions
*Core
FIX: (#6073) Grid: BDII make ldapsearch call forward compatible with newer ldapsearch versions
*Elasticsearch
NEW: (#6067) option op_type in index API for Elasticsearch
[v7r3p21]
*Resources
FIX: (#6059) use ceType instead of ceName for CE parameters
FIX: (#6051) string format for future in PoolCE
FIX: (#6013) HTCondorCE: make sure proxy is available when running condor_rm
NEW: (#5988) add CloudComputingElement
*WorkloadManagementSystem
FIX: (#6054) DIRACBenchmark client TypeError
FIX: (#6049) In case of a pilot version not matching the production version, the JobAgent expects the error
message to contain the words "Pilot version does not match", so the error message was updated accordingly.
FIX: (#6045) Don't crash if systemCall fails
FIX: (#6032) utf-8 support in job output for python2
CHANGE: (#5988) add hook for pilot cleanup
FIX: (#5979) StalledJobAgent can force status from Submitting to Failed
FIX: (#5979) ElasticJobParametersDB: make sure the docID is less than 512 characters
FIX: (#5946) escape time stamp in setHeartBeatData
*RequestManagementSystem
FIX: (#6020) if the request ID is an integer the test `is digit()` throws an exception
*Core
FIX: (#6011) Allow rootPath to be overridden with DIRAC_ROOT_PATH environment variable
*docs
FIX: (#6010) Added (renewed) jobs and pilot State Machines drawings
*DataManagementSystem
NEW: (#6008) BringOnline setting for FTS transfers can be SE dependant
[v7r3p20]
FIX: (#5977) allow to run Watchdog on non x86 platforms
*Core
FIX: (#5975) Limit reoccurrences of subprocess unicode error
FIX: (#5957) fix: allow for agents to loop every 10 seconds
FIX: (#5948) Revert "moved initialization of monitoring after agent initialize()"
*WorkloadManagementSystem
FIX: (#5972) update PoolCE.InnerCESubmissionType from JobAgent
FIX: (#5962) Stop fetching jobs when no more slot available in the inner PoolCE
FIX: (#5962) Make sure dirac-jobexec does not use the server certificate to execute the workflow
FIX: (#5957) can't change the polling time of JobAgent from inside an already running agent
FIX: (#5946) escape time stamp in setHeartBeatData
*DataManagementSystem
FIX: (#5967) fix a race condition when running multiple FTS3Agents in parallel
CHANGE: (#5924) DFC dump feature can dump more than one SE at the same time
*RequestManagementSystem
FIX: (#5953) dirac-rms-request takes into account PEP-515
FIX: (#5947) fix variable referenced before assignment in RequestTask
FIX: (#5947) take into account that targetSEList can be empty in the operations
*WorkloadManagement
CHANGE: (#5939) dirac-wms-get-cpu-normalization depends on db12 multiple_dirac_benchmark
*docs
FIX: (#5928) added better docs for JobShare and priorities
[v7r3p19]
*RequestManagementSystem
FIX: (#5929) filter by SourceSE in ReplicateAndRegister preparation
FIX: (#5929) Change RMS.Operation sourceSEList and targetSEList default to empty list
*Resources
FIX: (#5921) File protocol listDirectory returns metadata
CHANGE: (#5918) GFAL2_StorageBase redefine ECOMM if it does not exist
*WorkloadManagementSystem
FIX: (#5910) using a DErrno for no match found
FIX: (#5906) Matcher: fixed parsing of (required) tags
[v7r3p18]
*Core
CHANGE: (#5887) Use importlib.metadata rather than __RCSID__ for getting AgentModule version
CHANGE: (#5869) Show callstack in service logs when returning S_ERROR
*WorkloadManagementSystem
NEW: (#5884) Added Account option for SLURM
CHANGE: (#5850) HeartBeatTime is set every time a status is changed by the job itself
*Accounting
CHANGE: (#5871) decrease some logs verbosity
*ResourceStatusSystem
FIX: (#5863) return S_OK/S_ERROR in PublisherHandler.getSite
*WorkloadManagement
FIX: (#5859) add proxy in PushJobAgent and enhance it
*docs
CHANGE: (#5859) add more details about the PushJobAgent configuration
NEW: (#5852) Added general service configuration options description including Authorization options
FIX: (#5851) update DFC components description
CHANGE: (#5847) moved docs of ComputingElement from ConfReference to dirac.cfg
*DataManagementSystem
FIX: (#5853) FileCatalogCLI - removed faulty evaluation of undefined option to the rebuild command. Fixes #5759
[v7r3p17]
*WorkloadManagementSystem
CHANGE: (#5843) added Modules and PipInstallOptions to SiteDirector/Pilot option
FIX: (#5841) WMS DBs: do not try to decode when it's not bytes
*ConfigurationSystem
FIX: (#5839) VOMS2CSAgent correctly takes into account multiple CAs for users with multiple DNs
[v7r3p16]
*Core
CHANGE: (#5831) JEncode bytes, tests
FIX: (#5808) do not allow arbitrary code execution in JEncode
*ConfigurationSystem
CHANGE: (#5831) align with JEncode changes
*WorkloadManagementSystem
FIX: (#5815) Allow transition Rescheduled->Failed
FIX: (#5719) better use the job state machine when setting the job status, in particular when it comes from failover requests that may be inserted between 2 successful updates.
*Interface
FIX: (#5814) send only single jobID to checkJobStateTransition
*WorkloadManagement
CHANGE: (#5813) enable logs from db12 in dirac-wms-cpu-normalization
*DataManagementSystem
FIX: (#5790) Fix pickle error in dirac-dms-directory-sync
*TransformationSystem
Fix: Restore task state "Scheduled"
*docs
CHANGE: (#5812) add further details about the server installation process with python3
*FrameworkSystem
CHANGE: (#5760) added a flag for disabling the use of SecurityLogging service
[v7r3p15]
CHANGE: (#5761) use Client instead of RPCClient
*Resources
NEW: (#5755) allow to overwrite input/output storage plugin parameter with the CS
CHANGE: (#5755) add "xroot" to the list of input protocol for the XROOT plugin
CHANGE: (#5716) SingularityComputingElement: install python3 version
*Transformation
FIX: (#5753) dirac_production_runjoblocal.py: moved to py3 Pilot install
*Interfaces
CHANGE: (#5752) DIRAC APIs check if a JobState transition is allowed
*WorkloadManagementSystem
FIX: (#5749) JobSanity doesn't handle input sanboxes named "LFN:"
FIX: (#5745) convert inputs from tuple to list in JobAgent before submitting to a CE
FIX: (#5744) allow Stalled, Running and Matched jobs to go to Reschedule
FIX: (#5744) JobAgent forces the reschedule of jobs in case of exceptions
*HTTPS
FIX: (#5741) Replace DIRAC.Core.DISET.TransferClient with DIRAC.Core.Tornado.Client.ClientSelector
[v7r3p14]
*WorkloadManagementSystem
FIX: (#5732) securityLogging: zip after 1 day
NEW: (#5330) PushJobAgent to prepare jobs for sites with no external connectivity
NEW: (#5330) RemoteRunner to execute applications remotely (submitting them to remote CE)
*FrameworkSystem
FIX: (#5729) Get Tornado's port from the CS when adding URLs
*ResourceStatusSystem
FIX: (#5708) CSHelpers: some sites have no resources
*Resources
CHANGE: (#5330) add inputs and outputs parameters in ARC.submitJob()
*Workflow
CHANGE: (#5330) integrate RemoteRunner
[v7r3p13]
*WorkloadManagementSystem
FIX: (#5721) ElasticJobParametersDB: get in output all entries
CHANGE: (#5707) Replace DB12.py with the db12 package
CHANGE: (#5705) JobAgent and JobWrapper: only reporting changes on major status when needed
*Core
FIX: (#5720) TornadoBaseClientcorrectly checks CA
FIX: (#5715) encodeDict when the dict keys have mixed types
*HTTPS
FIX: (#5718) Add URLs to CS when installing HTTPS services
*ConfigurationSystem
FIX: (#5700) fix an exception in the VOMS2CSAgent when a user joins a second VO
[v7r3p12]
*ConfigurationSystem
CHANGE: (#5680) Bdii2CSAgent: add InjectSingleCoreQueue option to automatically create single core equivalents for MutliCore Queues , kind of fixes #5582
CHANGE: (#5680) Bdii2CSAgent: CEs in the BannedCEs list are no longer updated by the agent, previously this list only concerned CEs that are not already in the Configuration, fixes #5224
CHANGE: (#5680) dirac-admin-add-resources: add --onecore option to automatically create single core equivalents for MutliCore Queues
FIX: (#5680) dirac-admin-add-resources: fix the query about adding new CEs, this can now be answered in the negative
*Core
FIX: (#5655) DIRAC.isPy3VersionNumber always returns True
FIX: (#5653) Tasks never get removed from the ExecutorState if an UnrecoverableTaskException is raised
*Resources
FIX: (#5669) decode the http response in WLCGAccountingHTTPJson
*TransformationSystem
FIX: (#5661) fix RequestTasks flaky tests
*WorkloadManagement
FIX: (#5659) Python 3 support in SSHComputingElement
*RMS
FIX: (#5650) If a job was Killed, set it Killed at request completion
*DataManagementSystem
FIX: (#5649) dirac-dms-directory-sync: The script will now try all files and not end
a thread if one of the files assigned to be uploaded or downloaded to a thread fails.
The script will also print out all failed files, and their error at the end of a thread.
A few log messages have been corrected as well.
*FrameworkSystem
FIX: (#5622) SystemAdministratorCLI: get DIRAC version from Extensions
[v7r3p11]
*Resources
CHANGE: (#5646) ParallelLibrary: Embed the executable in the srun wrapper
*WorkloadManagement
FIX: (#5639) Bad error handling in OptimizationMindHandler.exec_taskError
*FrameworkSystem
CHANGE: (#5636) proxy expiration emails are sent from an address taken from
the Service/Agent config instead of the DB
*ResourceStatusSystem
FIX: (#5631) specifying the gocDB type for known types
*TransformationSystem
FIX: (#5628) TransformationCleaningAgent 'deletes', do not 'remove' jobs
NEW: (#5547) Allow MultiOperation body to fetch values from the task dict
NEW: (#5547) Introduce Body Plugins for DMS transformations
[v7r3p10]
*TransformationSystem
FIX: (#5628) TransformationCleaningAgent 'deletes', do not 'remove' jobs
FIX: (#5611) fix bug in TransformationCleaningAgent, that was NOT removing jobs but only deleting them
*Core
CHANGE: (#5618) use DIRAC_DEBUG_M2CRYPTO to enable SSL debugging in tornado
FIX: (#5590) Extension ordering in DIRACScript
*FrameworkSystem
FIX: (#5614) continue on empty service list in getComponentsStatus
FIX: (#5601) Cleaning old installations from a Python 3 based server
FIX: (#5600) Issue installing DIRAC and extensions on servers running Python 3
*RequestManagementSystem
NEW: (#5613) add SweepSize option to the ReqProxy
*WorkloadManagementSystem
FIX: (#5611) removing jobs instead of deleting them when bulk submission fails
FIX: (#5606) Return error if JobPolicy.getControlledUsers returns an empty list
FIX: (#5597) Don't run JobsStateMachine if newStat is ''
FIX: (#5593) UnicodeDecodeErrors in JobDB.getJobParameters
*ConfigurationSystem
FIX: (#5607) ServiceInterface.py - use ThreadPoolExecutor instead of ThreadPool to avoid thread leaks in the ConfigurationServer
CHANGE: (#5591) do not add the master CS first in the list of CS when new slaves are added
[v7r3p9]
*FrameworkSystem
FIX: (#5577) display correctly the help in sysadmin-cli
*WorkloadManagement
FIX: (#5565) Various fixes to executor infrastructure
FIX: (#5584) Error logging when optimisation fails
*tests
FIX: (#5576) fix typo in the HTTPs Resources tests
*Resources
FIX: (#5578) cloudinit-template - allow time in the monitor for the pilot to start
FIX: (#5578) cloudinit-template - set DIRACSYSCONFIG to point to the pilot.cfg in order to allow its use in the user applications
[v7r3p8]
CHANGE: (#5544) do not calculate FQDN unnecessarily
*WorkloadManagementSystem
CHANGE: (#5564) JobParameters - generalize getting CS parameters for Cloud VMTypes also
FIX: (#5550) PilotCStoJSONSynchronizer: do not look anymore in releases.cfg
*Core
FIX: (#5560) Fix overwriting files from an X509Chain object
*docs
CHANGE: (#5556) Forces docutils to 0.17 because of bug with sphynx Integration tests are fragile
and take resources and a lot of time and I don't see the need for them when changing
documentation or README for example. So I think it would be wise to turn them off in these cases.
*CI
CHANGE: (#5549) don't run Integration tests for documentation changes
[v7r3p7]
*Core
CHANGE: (#5536) Do not return failures when sending tasks to executors
NEW: (#5525) Utility for parsing DIRAC version number to Py3
FIX: (#5517) wait that the CS is loaded before checking for master CS in tornado-start-all
FIX: (#5515) Fix finding Dwatermark.png for the accounting watermark
*FrameworkSystem
CHANGE: (#5536) ComponentInstaller: look also for HTTPs services
NEW: (#5525) dirac-admin-update-instance converts version number to PEP-440 style if needed
FIX: (#5483) Allow Python 3 style pre-releases to be installed if the primary extension is a pre-release
*ConfigurationSystem
FIX: (#5534) VOMS2CSAgent: VOMS users that have multiple DNs and are suspended should no longer lead
to "User not registered" errors and have their status correctly set
*WorkloadManagementSystem
FIX: (#5527) old OptimizerModule can commit to ElasticJobParametersDB
FIX: (#5508) QueueUtilities - do not stop queue instantiation in case of failures
FIX: (#5503) SiteDirector - do not add dirac-install to the pilot sandbox
FIX: (#5503) JobStateUpdate - make setJobStatusBulk compatible with older clients
FIX: (#5500) CloudDirector - create only one pilot reference
FIX: (#5496) added another hack for JobLoggingDB
FIX: (#5490) rescheduling a job effectively sets the status to 'Received': force update
FIX: (#5515) Ensure valenc is bytes before decoding in retrieveOptimizerParam
*Resources
CHANGE: (#5523) Use PilotManagerClient to get CE status instead of interrogating PilotAgentsDB in HTCondorCE
FIX: (#5508) ComputingElement - remove unnecessary warning
FIX: (#5500) OpenStackCE - removed buggy line
NEW: (#5500) Utilities - added possibility to specify extra yum installable packages in the configuration
NEW: (#5500) Utilities - added possibility to specify ssh connection to the VM in the configuration
FIX: (#5500) cloudinit.template - use VMType as the Queue analog
FIX: (#5500) dirac_resource_get_parameters - fixed bugs in the VMType getting logic
FIX: (#5485) converted non-breaking whitespace to normal whitespace in cloud template
*Interfaces
FIX: (#5520) Dirac.py - fix for JobMonitoring.getJobsSummary now returns a structure and not a string
*RequestManagementSystem
CHANGE: (#5519) getRequestFileStatus takes better into account cases of multiple operations on the same file
CHANGE: (#5519) physicalRemoval checker expects an LFN and not a PFN
*TransformationSystem
NEW: (#5512) Allow GroupSize parameter to be mutable
NEW: (#5512) Add setGroupSize to command transformation CLI
FIX: (#5450) PluginUtilities.transID is not cast to an int
[v7r3p5]
*docs
FIX: (#5435) Corrected some grammar mistakes and unclear wording in the documentation
*DataManagementSystem
FIX: (#5436) minor fixes for ConsistencyInspector when no LFNs are given
NEW: (#5409) FTS3 tornado Handler
*WorkloadManagement
FIX: (#5437) only halt VM if JobAgent finished
*FrameworkSystem
FIX: (#5436) SystemAdministratorHandler: download dirac-install from management repo if py2 server
*RequestManagementSyste
NEW: (#5409) ReqManager tornado Handler
[v7r3p4]
FIX: fixes from v7r2p28
*WorkloadManagement
CHANGE: (#5429) Use python3 in VMDIRAC instances
[v7r3p3]
*Resources
NEW: (#5396) Add Emies endpoint to the ARC CE
*docs
FIX: (#5424) some minor fixes for user documentation
[v7r3p2]
*WMS
CHANGE: (#5404) use raw.githubusercontent URL for github raw files requests
*TS
CHANGE: (#5395) RequestTaskAgent uses objectLoader for RequestTasks
CHANGE: (#5395) split RequestTasks and WorkflowTasks into distinct module
*docs
NEW: (#5218) Updated Python2 and Python3 server installation instructions
CHANGE: (#5228) Documenting /WebApp/StaticResourceLinkDir option
CHANGE: (#5404) use raw.githubusercontent URL for github raw files requests
FIX: (#5404) use working git links for scripts
[v7r3p1]
FIX: Tag to push the PyPi deployment
[v7r3]
NEW: (#5000) Initial support for Python 3 server installations
NEW: VMDIRAC separate project is merged into the core DIRAC project
CHANGE: (#5035) use registerArgument to register positional arguments for scripts
CHANGE: (#4715) removing env variable DIRAC_USE_NEWTHREADPOOL
FIX: (#5035) use DIRACScript instead of Script
FIX: (#5028) Replaced all the cases of BaseException use by Exception
*Core
NEW: (#5062) provide some docs and add registerArgument method to register arguments in Script
NEW: (#4997) dirac-configure can now be ran without arguments when using Python 3
CHANGE: (#4937) removed dirac-agent, dirac-service, dirac-executor scripts (use '_' counterparts instead)
CHANGE: (#5110) Removed Core.Utilities.Grid.getBdiiCEInfo function use Utilities.Glue2.getGlue2CEInfo.
Also dropped ldapSite, ldapCluster, ldapCE, ldapCEState, ldapCEVOView, ldapService functions
CHANGE: (#4903) Using former RSS State Machine as general State Machine
CHANGE: (#5237) Use generic ObjectLoader.loadObjects() function in the specific Plotting.ObjectLoader class
CHANGE: (#5244) dirac-configure: no upload attempt upon initial proxy generation
CHANGE: (#5246) Drop MySQL._to_string() method
CHANGE: Release constraint on SQLALchemy versions
FIX: (#5210) show extremely small values in pie plots
FIX: (#4997) Handle SIGINT correctly when reading certificate passwords
FIX: (#5269) Use isinstance when checking types in MessageFactory
FIX: (#5272) encode stub before base64
FIX: (#5326) fix bug in getSystemURLs: getOptionsFromCFG can return None
CHANGE: (#5337) removed src/DIRAC/Core/scripts/dirac-install.py
CHANGE: (#5372) RCSID is no longer access for Python 3 based installations
*Configuration
NEW: (#5062) add registerCmdArg to register arguments and group argument to group returned arguments
NEW: (#5286) test new PathFinder methods
NEW: (#5286) PathFinder - add checkServiceURL that check URL port and path
NEW: (#5286) PathFinder - add getServiceURLs that return list type result
NEW: (#5286) PathFinder - add getSystemURLs that return all services URLs for system
CHANGE: (#5110) Dropped Glue2Only and Glue2URLS options from Bdii2CSAgent, Glue2 is now the only way
CHANGE: (#5110) dirac-admin-add-resources: drop -g -G options, Glue2 is now the only way
CHANGE: (#5110) Configuration.Client.Utilities functions getGridCEs, getSiteUpdates no longer take glue2 parameter
CHANGE: (#5000) Deprecate CSGlobals.getInstalledExtensions, DIRAC.Core.Utilities.Extensions.extensionsByPriority
should be used instead
CHANGE: (#5000) Deprecate getCSExtensions, DIRAC.Core.Utilities.Extensions.extensionsByPriority should be used instead
CHANGE: (#5286) PathFinder - getSystemSection do not use serviceTuple, add docs, use system and service instead
of componentTuple
FIX: (#5286) PathFinder - getGatewayURLs pass list to randomize when serviceName is empty
FIX: (#5298) fix getServiceFailoverURL for multi url case
FIX: (#5375) Handle S_ERROR result in CSAPI, docs
*DMS
CHANGE: (#5206) prints why dirac-dms-add-files failed
*Framework:
CHANGE: (#4303) Removed completely the SystemLogging
CHANGE: (#5164) Remove HashTag table from Framework/UserProfile
CHANGE: (#5164) Remove dataTypeRE argument when retrieving values from Framework/UserProfileClient
FIX: (#5340) Fix using the System Administrator to install Python 3 releases of DIRAC from Python 2
*Interfaces
CHANGE: (#5110) DiracAdmin: dropped getBDII* functions, which were looking for Glue1 information
CHANGE: (#5110) removed dirac-admin-bdii-info command
*Resources
CHANGE: (#5314) Remove GlobusComputingElement
*RMS
CHANGE: (#5291) ReqDB don't cast datetime to strings
*RSS
NEW: (#5042) Multi-VO mode of operation support
FIX: (#5335) VO parameter is optional. When omitted commands will act on all VOs.
*WMS
NEW: (#4903) Added WMS (Jobs) State Machine
NEW: (#5214) Added a HTTPs JobManager service
NEW: (#5214) Added a HTTPs JobMonitor service
NEW: (#5214) Added a HTTPs JobStateUpdate service
NEW: (#5314) Add a PilotStatus module to clearly define the pilot status in the code
CHANGE: (#5214) Removed need for ThreadScheduler in JobManagerHandler
CHANGE: (#5289) SiteDirector: do not send dirac-install if it's py3 pilot
CHANGE: (#4884) JobDB: compress JDLs by default (no need for flag anymore)
CHANGE: (#4937) removed StatesMonitoringAgent (use StatesAccountingAgent agent instead)
CHANGE: (#5141) add an Aborted_HOUR column to the pilot efficiency table needed by the WebApp
CHANGE: (#5178) JobMonitoring does not need to look into TaskQueue
CHANGE: (#5246) remove pilots before start Test_PilotsClient test
CHANGE: (#5318) Move SiteDirector.getQueues in QueueUtilities module
FIX: (#5246) mock _escapeString in Test_Agent_PilotStatusAgent to pass test
FIX: (#5246) escape values in PilotAgentsDB
FIX: (#5246) decode values using decode() method in JobDB
NEW: (#5314) Add a PilotStatus module to clearly define the pilot status in the code
FIX: (#5325) Fix-up minor errors introduced during VMDIRAC merge
NEW: (#5325) Tests for VirtualMachine DB & Service
FIX: (#5342) Allow Platform parameter to be missing from the queue configuration
*TS
NEW: (#5327) Added TransformationFilesStatus module
*tests
NEW: (#5062) add Test_LocalConfiguration
CHANGE: (#5046) don't use mail in the self generated certificates
CHANGE: (#5301) added RALPP as a multi-VO test site and added env dump to job.log to ease debugging
NEW: (#5321) added integration test for AccountingDB
CHANGE: (#5362) Add proxy correct setup within wms-script.sh
*docs
CHANGE: (#5313) replace the theme by the sphinx_rtd_theme
[v7r2p31]
*ConfigurationSystem
FIX: (#5524) VOMS2CSAgent: VOMS users that have multiple DNs and are suspended should no longer lead to "User not registered" errors and have their status correctly set
*Core
CHANGE: (#5492) Do not return failures when sending tasks to executors
FIX: (#5474) Always get voms-proxy-init from DIRACOS to avoid issues generating proxies on CentOS 8
CHANGE: (#5461) Sleep instead of dropping requests when services are overloaded
FIX: (#5494) DErrno.py - do not attempt to add extra error definitions from DIRAC core
*FrameworkSystem
CHANGE: (#5492) ComponentInstaller: look also for HTTPs services
CHANGE: (#5510) SecurityLogClient - set security log message to DEBUG
*TransformationSystem
NEW: (#5509) Allow GroupSize parameter to be mutable
NEW: (#5509) Add setGroupSize to command transformation CLI
*WorkloadManagementSystem
CHANGE: (#5480) We have race conditions between the termination of jobs and
the failover request. There is also a problem with RAL storage as a failed
upload would remove the file recovered by failover if this is too soon.
Hence delay to 45 minutes the execution of failover requests
*RequestManagementSystem
CHANGE: (#5449) getRequestFileStatus takes better into account cases of multiple operations on the same file
CHANGE: (#5449) physicalRemoval checker expects an LFN and not a PFN
CHANGE: (#5495) ForwardDISET uses Client to be able to talk to https services
CHANGE: (#5438) delay the execution of a request in case the user has been suspended (6 hours for the time being)
*Resources
FIX: (#5487) Slurm getJobStatus split fields
*tests
CHANGE: (#5465) adapt pylintrc to black
[v7r2p30]
*Core
FIX: (#5474) Always get voms-proxy-init from DIRACOS to avoid issues generating proxies on CentOS 8
CHANGE: (#5461) Sleep instead of dropping requests when services are overloaded
*tests
CHANGE: (#5455) add sweep as keyword and case insensitive
*WorkloadManagementSystem
FIX: (#5462) wait an incremental amount of seconds for finding the child PID
FIX: (#5469) disconnect the executor from the ExecutorMind when the connection drops.
FIX: (#5442) Optimizers: fail job if ISB LFN does not start with '/'
*Resources
FIX: (#5439) Stomp MQ: reconnect in case of lost connection
*Test
CHANGE: (#5465) adapt pylintrc to black
*RequestManagementSystem
CHANGE: (#5438) delay the execution of a request in case the user has been suspended
(6 hours for the time being)
[v7r2p28]
*Core
FIX: (#5416) unless explicitly set, do not skip the CA checks
*WorkloadManagement
FIX: (#5431) Ensure invalid tasks don't get rescheduled by ExecutorDispatcher
NEW: (#5430) add support for execution of pilot wrappers when the python executable is not set
*RMS
FIX: (#5432) ReqClient uses correct credentials to talk to ReqProxies
FIX: (#5432) ReqClient uses proper WMS client insteand of RPCClient
*Resources
CHANGE: (#5428) asctime now includes microsecond precision when using the MessageQueueBackend for logging
*DMS
FIX: (#5427) adapt csv use to py3
*tests
FIX: (#5427) adapt csv use to py3
[v7r2p27]
*TS
CHANGE: (#5413) TransformationDB: moved GroupSize from INTEGER to FLOAT
*Resources
FIX: (#5421) fix Slurm getJobStatus() method
[v7r2p26]
*Core
FIX: (#5390) Handle S_ERROR in Service._executeAction
NEW: (#5145) Add returnValueOrRaise and convertToReturnValue to simplify writing code with exceptions
CHANGE: (#5394) Requests are now dropped if the backlog is too large
FIX: (#5411) RPCClient and TornadoClient support proxyChain
*Framework
FIX: (#5410) ProxyManagerClient - upload proxy without embedded DIRAC group
*WMS
FIX: (#5414) JobCleaningAgent uses OwnerDN to put the sandbox removal request
*DMS
CHANGE: (#5284) FTS DB change (see https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-v7r2#new-status-for-fts3-files-and-jobs)
NEW: (#5284) enable FTS archive monitoring
NEW: (#5284) allow to use FTS activity based on FTS3Plugin
FIX: (#5408) making use of freesize in WLCGAccountingJson if exists, instead of calculating
as Total - usedsize
*RMS
CHANGE: (#5284) ReplicateAndRegister prefers disk replicas
*Resources
NEW: (#5402) DIRAC_GFAL_GRIDFTP_ENABLE_IPV6 to control gridftp behavior
*TS
FIX: (#5405) Transformation.setBody correctly checks all the operation tuples
*tests
NEW: (#5403) force commit format
[v7r2p25]
FIX: (#5373) Fix un-returned use of S_ERROR and S_OK
*tests
CHANGE: (#5380) Update DIRACCAProxyProvider test to use pytest
FIX: (#5381) Adapt Test_FilePlugin.py for https://bugs.python.org/issue43219
*WMS
FIX: (#5380) Fix bad merge in JobMonitoringHandler.py when producing v7r2p24
[v7r2p24]
FIX: fixes from v7r1p46
*Resources
FIX: (#5267) return error for StompMQConnector
[v7r2p23]
*WMS
CHANGE: (#5348) PilotWrapper: added file:/cvmfs/dirac.egi.eu/pilot/ as location
CHANGE: (#5356) SiteDirector - continue queues initialization if one of them fails
*Resources
FIX: (#5356) executeBatch - fallback to urllib if six module is not available
FIX: (#5356) Host.py - return Message in case of non-zero status in getCEStatus
FIX: (#5356) SSHBatchComputingElement - added forgotten instantiation of the BatchSystem plugin
[v7r2p22]
*Workflow
FIX: (#5346) UnicodeEncodeError when application log contains non-ascii characters
*Resources
FIX: (#5345) Pass --cfg /tmp/pilot.cfg when reusing host DIRAC installation in SingularityComputingElement
[v7r2p21]
*Core
FIX: (#5334) Use selectors(2) instead of select.select to avoid issues with file descriptors > 1024, fixes #5232
FIX: (#5336) ElasticSearchDB: use ca certs file if requested
FIX: (#5328) ElasticSearchDB: work on a copy of the documents in generateDocs()
*WorkloadManagement
FIX: (#5332) Fix unassigning sandboxes when removing jobs from the transformation system
*Resources
NEW: (#5329) Added PBSResourceUsage unit test