feat(release): v2.6.0 (#650)

* fix: Add block reported user task (#614) * backport: remove sentry (#622) * bump: Module Spam Detection to 4.1.2 (#630) * bump: Fix geocofing on homepage interactive map (#635) * feat: Add module emitter (#633) * feat: Addition of the emitter module * fix: Fix failing specs * Fix/backport decidim awesome slowness on proposals index page (#631) * add env variable * add UUID and IP to logs * add weighted voting configuration * add secrets for weighted voting * update test to fix CI * continue fix test file * fix CI * fix CI * clean spec * clean and add test * backport: Addition of sortable scopes using drag and drop (#632) * backport: Backport the Drag & Drop on Scopes on BackOffice * fix: Fix locales that were not normalized or misplaced * test: Add some specs about the check_boxes_tree_helper sort * test: Add specs releated to the backport scopes management * fix: Fix wrong locales * fix: Add the weight sorting on select input of scopes * fix: Add missing locale * feat: Allow to choose notification settings when attachment added (#627) * fix: Override Create attachment admin command * fix: Force email notification on attachment event * fix: Override attachment form * feat: Allow admins to toggle notification sending * refactor: Remove override and add extend * lint: Fix rubocop offenses --------- Co-authored-by: Lucie Grau <[email protected]> * fix: Remove caching from the geocoding elements to avoid map not reloading when refreshing (#638) * backport: Reorder scopes in meetings (#639) * fix: Scopes can't be updated in BO (#640) Co-authored-by: Lucie Grau <[email protected]> * backport: Add layer of security and download p7zip-full lib on docker (#643) * bump: Bump custom proposal states (#599) * Feat: custom sort for processes (#596) * feat: add custom sort for processesdepending on new variable * feat: update locales files * feat: update seeds * test: add controller tests for assemblies and processes * chore: update i18n config for unused keys * docs: update overrides * refactor: update env variable after review * Fix form initiatives (#600) * fix: XSS vulnerability with img on initiative form and model * test: add tests for new validation * docs: update overrides section * fix: interference from added extends with migration * style: update with rubocop * fix: ActiveRecord::NoDatabaseError * fix: trying to fix again interference * fix: update initiative fomr extends and modify admin initiative controller * refactor: update with rubocop * fix: validation in initiative_form extends and update test * docs: update overrides section in overloads.md * fix: Update OVERLOADS.md --------- Co-authored-by: Quentin Champenois <[email protected]> * bump: Decidim-Awesome to last commit (#607) * feat: Bump decidim-awesome to last commit * fix: Fix migration that has been changed since first implementation * fix: Add Referrer-Policy to strict (#613) * fix: Flash message on proposal limit per user reached (#609) * fix error message displaying when reaching proposition add limit * add test * add keys in ignore_missing keys * fix: Add block reported user task (#614) * feat: Add module decidim-cleaner (#597) * feat: Add decidim-cleaner * refactor: Comment env var by default --------- Co-authored-by: Quentin Champenois <[email protected]> * Install GuestMeetingRegistration module (#615) * Install GuestMeetingRegistration module * Update registration module * feat: Add Sendethics possibility to the sms gateway (#605) Co-authored-by: Lucie Grau <[email protected]> * Bump: Phone authorization handler module (#623) * backport: remove sentry (#622) * backport: Use cdn (#624) * backport: self hosted cdn * backport: update js files content * fix: Missing image in survey question (#621) * fix: update condition to not empty input value if image is present * test: add system test to check for input value * test: update check for image * test: update img check again * test: last update check img * test: update * test: another update * test: if img is present * test: update other test to avoid ambiguous selector error * test: update to see if image is presnet after save * fix: override editor js in decidim_awesome * test: update system test * fix: Questions order in survey export (#618) * chore: update after pull * fix: order questions by position in serializer * test: add test for question order * refactor: update test * style: remove empty line * bump: Guest meeting registration module (#625) * Bump: Guest meeting registration module * refactor(Gemfile): Remove ref reference in Gemfile --------- Co-authored-by: Quentin Champenois <[email protected]> * feat: add decypted private body to extra fields (#608) * feat: add new column to proposal extra fields * feat: add callback to proposal extra field model * test: add test for proposal extra field model * feat: add rake task to update existing data * test: add test for new rake task * feat: add proposal extra field model extends to config * chore: update rubocop rules * refactor: update task and test * create the job file * update rake task * lint code * add tests file * fix rspec * clean double specs * update spec * update spec * update syntax test with a context instade of only 'it' * lint code by removing useless line in job spec file * update spec * add more context in spec file * lint code * update rake tasks test --------- Co-authored-by: barbara oliveira <[email protected]> Co-authored-by: Lucie Grau <[email protected]> Co-authored-by: Quentin Champenois <[email protected]> * feat: Clear minio s3 bucket (#612) * feat(Docker): Add minio service * feat(rake): Add new tasks to cleanup s3 bucket * fix: Add S3 purge rake task * fix: S3 Bucket endpoint for docker local * fix(rake): Active storage clear orphans job * fix(sidekiq): Add sidekiq configuration * fix: Logger for active_storage.rake job * fix: Prevent duplicated ActiveRecord Query * fix: Prevent error on PP sort with end_date nil (#626) * fix: update to handle processes without start_date or end_date * fix: update sort in controllers * test: update test with process without end date * refactor: optimize queries in controllers and update tests * refactor: update sort processes in controllers --------- Co-authored-by: Lucie Grau <[email protected]> * bump: Module Spam Detection to 4.1.2 (#630) * feat: Author notification on proposal publication (#620) * add notification with eventmanager * base to watch the CI and see files on github * fix translation key & notififaction displaying * add send_pubication_notification to right file * start test rspec * continuing rspec * potential final test file * fix: Merge proposal command and anonymous proposals * fix: Change ProposalPublishedEvent to SimpleEvent * fix: Proposal Published Event * fix: Push FR locales * test: Add specs for proposal_published_event * update methode & file name * adjust trad key * adjust test file & update name of test file * lint * lint fr trad key * lint * update trad key order * lint * correct trad key link in method * correst rspec * fix rspec * fix: Notification small title * lint(rubocop): Fix offenses * ci: Exclude BeEq Rubocop rule * clean * fix CI * delete test file * add ignore trad key in i118n-tasks.yml to fix CI * update text syntaxe via trad key * add '' in fr trad key --------- Co-authored-by: Quentin Champenois <[email protected]> * revert: "fix: Flash message on proposal limit per user reached (#609)" (#634) This reverts commit 28003b5. * bump: Fix geocofing on homepage interactive map (#635) * feat: Add module emitter (#633) * feat: Addition of the emitter module * fix: Fix failing specs * Fix/backport decidim awesome slowness on proposals index page (#631) * add env variable * add UUID and IP to logs * add weighted voting configuration * add secrets for weighted voting * update test to fix CI * continue fix test file * fix CI * fix CI * clean spec * clean and add test * backport: Addition of sortable scopes using drag and drop (#632) * backport: Backport the Drag & Drop on Scopes on BackOffice * fix: Fix locales that were not normalized or misplaced * test: Add some specs about the check_boxes_tree_helper sort * test: Add specs releated to the backport scopes management * fix: Fix wrong locales * fix: Add the weight sorting on select input of scopes * fix: Add missing locale * feat: Allow to choose notification settings when attachment added (#627) * fix: Override Create attachment admin command * fix: Force email notification on attachment event * fix: Override attachment form * feat: Allow admins to toggle notification sending * refactor: Remove override and add extend * lint: Fix rubocop offenses --------- Co-authored-by: Lucie Grau <[email protected]> * fix: Remove caching from the geocoding elements to avoid map not reloading when refreshing (#638) * backport: Reorder scopes in meetings (#639) * fix: Scopes can't be updated in BO (#640) Co-authored-by: Lucie Grau <[email protected]> * backport: Add layer of security and download p7zip-full lib on docker --------- Co-authored-by: stephanierousset <[email protected]> Co-authored-by: Quentin Champenois <[email protected]> Co-authored-by: ’Barbara Oliveira <[email protected]> Co-authored-by: Quentin Champenois <[email protected]> Co-authored-by: Alexandru Emil Lupu <[email protected]> Co-authored-by: Lucie Grau <[email protected]> Co-authored-by: barbara oliveira <[email protected]> * fix(smtp): Add authentication plain text only if user_name & password (#645) * fix: Fix characters limits when editing comments (#646) * fix: Fix the editing of comments to put the same chars limit than it has on its creation * fix: Add missing locales * fix: Clear papertrail versions in database (#647) * fix(job): Papertrail versions job * fix(rake): Create rake task and sidekiq task * fix(rake): Add log information * fix(rake): Add env var to rake task * fix(job): Add versions to remove * refactor: Rename variable expiration to retention * refactor: Use Rails secrets * fix: Add DB size in system view * fix(system): Add Decidim version in system view * fix(job): Default batch size to 5000 * fix: System dashboard locales * fix: Ensure coordinates from address field are not replaced (#648) Co-authored-by: Quentin Champenois <[email protected]> * feat: Alphabetical order addition for Budgets' Projects (#652) * feat: Addition of alphabetical order on Budgets Projects * feat: Addition of the default order on the Budgets' BackOffice * feat: Add missing specs about budget's component * fix: Fix wrong translation * Fix/popup thumbnail on map (#654) * feat: add leaflet-tilelayer-here package * fix: update options of popup --------- Co-authored-by: Lucie Grau <[email protected]> --------- Co-authored-by: stephanierousset <[email protected]> Co-authored-by: Guillaume MORET <[email protected]> Co-authored-by: ’Barbara Oliveira <[email protected]> Co-authored-by: Lucie Grau <[email protected]> Co-authored-by: Alexandru Emil Lupu <[email protected]> Co-authored-by: barbara oliveira <[email protected]>
OpenSourcePolitics · Jan 16, 2025 · b8bbb21 · b8bbb21
1 parent be5f07d
commit b8bbb21
Show file tree

Hide file tree

Showing 32 changed files with 9,374 additions and 7,408 deletions.
diff --git a/Gemfile.lock b/Gemfile.lock
@@ -1235,4 +1235,4 @@ RUBY VERSION
    ruby 3.0.6p216
 
 BUNDLED WITH
-   2.5.22
+   2.5.10
diff --git a/app/controllers/concerns/decidim/budgets/orderable.rb b/app/controllers/concerns/decidim/budgets/orderable.rb
@@ -0,0 +1,94 @@
+# frozen_string_literal: true
+
+require "active_support/concern"
+
+module Decidim
+  module Budgets
+    # Common logic to sorting resources
+    module Orderable
+      extend ActiveSupport::Concern
+
+      included do
+        include Decidim::Orderable
+
+        private
+
+        # Available orders based on enabled settings
+        def available_orders
+          @available_orders ||= [default_order] + possible_orders.excluding(default_order)
+        end
+
+        def possible_orders
+          @possible_orders ||= begin
+            available_orders = []
+            available_orders << "random" if voting_open? || !votes_are_visible?
+            available_orders << "most_voted" if votes_are_visible?
+            available_orders += %w(alphabetical highest_cost lowest_cost)
+            available_orders
+          end
+        end
+
+        def default_order
+          @default_order ||= fetch_default_order
+        end
+
+        def fetch_default_order
+          default_order = current_settings.default_sort_order.presence || component_settings.default_sort_order
+          return order_by_default if default_order == "default"
+
+          possible_orders.include?(default_order) ? default_order : order_by_default
+        end
+
+        def order_by_default
+          voting_open? || !votes_are_visible? ? "random" : "most_voted"
+        end
+
+        def votes_are_visible?
+          current_settings.show_votes?
+        end
+
+        def reorder(projects)
+          case order
+          when "alphabetical"
+            reorder_alphabetically(projects)
+          when "highest_cost"
+            reorder_by_highest_cost(projects)
+          when "lowest_cost"
+            reorder_by_lowest_cost(projects)
+          when "most_voted"
+            reorder_by_most_voted(projects)
+          when "random"
+            reorder_randomly(projects)
+          else
+            projects
+          end
+        end
+
+        def reorder_alphabetically(projects)
+          projects.ordered_ids(
+            projects.sort_by { |project| project.title[I18n.locale.to_s] || "" }.map(&:id)
+          )
+        end
+
+        def reorder_by_highest_cost(projects)
+          projects.order(budget_amount: :desc)
+        end
+
+        def reorder_by_lowest_cost(projects)
+          projects.order(budget_amount: :asc)
+        end
+
+        def reorder_by_most_voted(projects)
+          return projects unless votes_are_visible?
+
+          ids = projects.sort_by(&:confirmed_orders_count).map(&:id).reverse
+          projects.ordered_ids(ids)
+        end
+
+        def reorder_randomly(projects)
+          projects.order_randomly(random_seed)
+        end
+      end
+    end
+  end
+end
diff --git a/app/controllers/decidim/system/dashboard_controller.rb b/app/controllers/decidim/system/dashboard_controller.rb
@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module Decidim
+  module System
+    class DashboardController < Decidim::System::ApplicationController
+      before_action :check_organizations_presence
+
+      def show
+        @organizations = Organization.all
+        @db_size = db_size
+      end
+
+      def check_organizations_presence
+        return if Organization.exists?
+
+        redirect_to new_organization_path
+      end
+
+      private
+
+      def db_size
+        dbname = ActiveRecord::Base.connection.current_database
+        sql = "SELECT pg_size_pretty(pg_database_size('#{dbname}'));"
+        ActiveRecord::Base.connection.execute(sql)[0]["pg_size_pretty"]
+      end
+    end
+  end
+end
diff --git a/app/jobs/clear_duplicated_half_signup_users_job.rb b/app/jobs/clear_duplicated_half_signup_users_job.rb
@@ -0,0 +1,127 @@
+# frozen_string_literal: true
+
+class ClearDuplicatedHalfSignupUsersJob < ApplicationJob
+  include Decidim::Logging
+
+  def perform
+    @dup_decidim_users_count = 0
+    @dup_half_signup_count = 0
+
+    log! "Start clearing half signup accounts..."
+    if duplicated_phone_numbers.blank?
+      log! "No duplicated phone numbers found"
+      return
+    end
+
+    log! "Found #{duplicated_phone_numbers.count} duplicated phone number to cleanup"
+    duplicated_phone_numbers.each do |phone_info|
+      phone_number, phone_country = phone_info
+      users = Decidim::User.where(phone_number: phone_number, phone_country: phone_country)
+
+      clear_data users
+    end
+
+    log! "Total distinct numbers to clear : #{duplicated_phone_numbers.size}"
+    log! "Half signup users archived : #{@dup_half_signup_count}"
+    log! "Decidim users account updated : #{@dup_decidim_users_count}"
+    log! "Total accounts modified : #{@dup_half_signup_count + @dup_decidim_users_count}"
+    log! "Terminated !"
+  end
+
+  private
+
+  def duplicated_phone_numbers
+    @duplicated_phone_numbers ||= Decidim::User
+                                  .where.not(phone_number: [nil, ""])
+                                  .where.not(phone_country: [nil, ""])
+                                  .group(:phone_number, :phone_country)
+                                  .having("count(*) > 1")
+                                  .pluck(:phone_number, :phone_country)
+  end
+
+  def clear_data(users)
+    decidim_user_dup_accounts = []
+
+    users.each do |user|
+      if user.email.include?("quick_auth")
+        @dup_half_signup_count += 1
+        soft_delete_user(user, delete_reason)
+      else
+        @dup_decidim_users_count += 1
+        decidim_user_dup_accounts << user
+      end
+    end
+
+    return if decidim_user_dup_accounts.blank?
+    # The unique user might be a user without email, if so, it should be cleared
+    return if decidim_user_dup_accounts.size <= 1 && decidim_user_dup_accounts.first.email.present?
+
+    # if there is multiple decidim user accounts, clear all phone number for these accounts
+    decidim_user_dup_accounts.each do |decidim_user|
+      clear_account_phone_number(decidim_user)
+    end
+  end
+
+  def soft_delete_user(user, reason)
+    return unless user.email&.include?("quick_auth")
+
+    email = user.email
+    phone = user.phone_number
+    user.extended_data = user.extended_data.merge({
+                                                    half_signup: {
+                                                      email: email,
+                                                      phone_number: phone,
+                                                      phone_country: user.phone_country
+                                                    }
+                                                  })
+
+    user.phone_number = nil
+    user.phone_country = nil
+
+    form = Decidim::DeleteAccountForm.from_params(delete_reason: reason)
+    Decidim::DestroyAccount.call(user, form) do
+      on(:ok) do
+        log!("User (ID/#{user.id} email/#{email} phone/#{obfuscate_phone_number(phone)}) has been deleted")
+      end
+      on(:invalid) do
+        log!("User (ID/#{user.id} email/#{email} phone/#{obfuscate_phone_number(phone)}) cannot be deleted: #{form.errors.full_messages}")
+      end
+    end
+  end
+
+  def clear_account_phone_number(user)
+    phone_number = user.phone_number
+    Decidim::User.transaction do
+      user.extended_data = user.extended_data.merge({
+                                                      half_signup: {
+                                                        phone_number: user.phone_number,
+                                                        phone_country: user.phone_country
+                                                      }
+                                                    })
+
+      user.phone_number = nil
+      user.phone_country = nil
+      user.save(validate: false)
+    end
+
+    log! "User (ID/#{user.id} phone/#{obfuscate_phone_number(phone_number)} email/#{user.email}) has been cleaned"
+  end
+
+  def obfuscate_phone_number(phone_number)
+    return "No phone number" if phone_number.blank?
+
+    visible_prefix = phone_number[0..1]
+    visible_suffix = phone_number[-2..]
+    obfuscated_middle = "*" * (phone_number.length - 4)
+
+    visible_prefix + obfuscated_middle + visible_suffix
+  end
+
+  def current_date
+    Date.current.strftime "%Y-%m-%d"
+  end
+
+  def delete_reason
+    "HalfSignup duplicated account (#{current_date})"
+  end
+end
diff --git a/app/jobs/concerns/decidim/logging.rb b/app/jobs/concerns/decidim/logging.rb
@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module Decidim
+  module Logging
+    private
+
+    def log!(msg, level = :warn)
+      msg = "(#{self.class})> #{msg}"
+
+      case level
+      when :info
+        Rails.logger.info msg
+        stdout_logger.info msg unless Rails.env.test?
+      else
+        Rails.logger.warn msg
+        stdout_logger.warn msg unless Rails.env.test?
+      end
+    end
+
+    def stdout_logger
+      @stdout_logger ||= Logger.new($stdout)
+    end
+  end
+end
diff --git a/app/jobs/decidim/papertrail_versions_job.rb b/app/jobs/decidim/papertrail_versions_job.rb
@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+module Decidim
+  class PapertrailVersionsJob < ApplicationJob
+    queue_as :default
+
+    include Decidim::Logging
+
+    def perform(ret = nil)
+      ret = retention(ret)
+
+      log! "Cleaning versions in database..."
+      log! "Cleaning item_types : #{item_types.join(", ")}"
+
+      total = 0
+      PaperTrail::Version.where(item_type: item_types).where("created_at <= ?", ret).in_batches(of: 5000) do |versions|
+        total += versions.size
+        versions.destroy_all
+      end
+
+      log! "#{total} versions removed"
+    end
+
+    private
+
+    def retention(ret)
+      return ret if ret.present? && ret.is_a?(Time)
+
+      ret = Rails.application.secrets.dig(:decidim, :database, :versions, :clean, :retention)
+      ret.months.ago
+    end
+
+    # Exhaustive list of item_types to remove from versions table
+    def item_types
+      @item_types ||= %w(
+        Decidim::Accountability::TimelineEntry
+        Decidim::Accountability::Result
+        Decidim::Attachment
+        Decidim::AttachmentCollection
+        Decidim::Blogs::Post
+        Decidim::Budgets::Project
+        Decidim::Comments::Comment
+        Decidim::Conferences::MediaLink
+        Decidim::Conferences::Partner
+        Decidim::Debates::Debate
+        Decidim::Categorization
+        Decidim::Categorization
+        Decidim::Forms::Questionnaire
+        Decidim::UserBaseEntity
+      )
+    end
+  end
+end
-Original file line number
+Diff line change
@@ Expand Up / @@ -1235,4 +1235,4 @@ RUBY VERSION @@
        ruby 3.0.6p216
     BUNDLED WITH
-.5.22
+.5.10