A method for constructing effective Hamiltonian in the subspace of the functions total angular momentum F (including the nuclear spin I) is considered in the paper. The effective Hamiltonian is presented for a diatomic molecule having the nuclear spin I = 5/2 in the state 3Σ. The calculational results for isotopic modifications 16O17O and 17O18O of the oxygen molecule based on the Hamiltonian obtained are analyzed. The reliability of the effective Hamiltonian formulation proposed was checked by a comparison of line centers computed with the experimental ones and it was assessed to be satisfactory.